# Using a JuMP \`Bin\` vector to index an array (checkbounds fails with type VariableRef)

**URL:** <https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674>\
**Category:** Optimization (Mathematical)\
**Tags:** question\
**Created:** [August 10, 2023, 11:46am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674 "2023-08-10T11:46:19Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 10, 2023, 11:46am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/1 "2023-08-10T11:46:20Z")

</div>

Hi there, I am following up from an earlier post [[How to "prioritize" choosing some parameters over others in Hyperopt.jl or similar packages](https://discourse.julialang.org/t/how-to-prioritize-choosing-some-parameters-over-others-in-hyperopt-jl-or-similar-packages/102594) ] where I was instructed to use JuMP with decision variables.

I have a simple MWE, where I try to optimize the **selection** of the columns of a matrix, so that the columns sum to a minimum number.

```julia
using JuMP
using HiGHS

# matrix with 20 columns:
m = rand(1000, 20)

# My objective: find the columns of the matrix that sum to the least
# i.e., given a boolean vector with trues meaining what to choose, minimize:

function objective(chosen)
    r = m[:, chosen]
    q = sum(sum(c) for c in eachcol(r))
    return q
end
# random choice of chosen:
chosen = falses(20)
chosen[[1, 5, 6]] .= true

objective(chosen)

# Attempt with Jump, following the Sudoku tutorial
model = Model(HiGHS.Optimizer)
# The variable chosen is a boolean vector with
# the `true` entries meaning a choice that column
@variable(model, choices[1:20], Bin);
# Constrain the amount of columns that can be chosen
Fmin = 2
Fmax = 3
@constraint(model, Fmin ≤ sum(choices) ≤ Fmax)
@objective(model, Min, objective(choices))

# errors

```

This example errors because the last line that calls `@objective` yields: `ERROR: ArgumentError: unable to check bounds for indices of type VariableRef`.

I’ve searched around in the JuMP docs and even though I found examples similar to the one I need (optimizing recipes for example) I couldn’t find something where the decision variables are used to index an “external” array.

What is the solution I am missing here?

p.s.: In my real use case the `objective` function is extremely complicated and cannot be expressed in terms of mathematical equations. Nevertheless, it only needs to access the columns of a pre-existing matrix in such a way.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 10, 2023, 11:55am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/2 "2023-08-10T11:55:48Z")

</div>

I don’t think you can index directly, but often you don’t need to. For example, you can use them as a mask, e.g.

```julia
m = rand(1000, 20)
chosen = rand(Bool, 20) # not the actual variables, just to show
column_sums = [sum(c) for c in eachcol(m)]
selected_sums = column_sums .* chosen # set non-chosen sums to 0
sum(selected_sums) # final sum of selected columns

```

The reason why you can’t use them directly as indices (as I understand it) is that to solve the whole problem, solvers will often solve relaxations of the problem in which the integer constraints are lifted, meaning those variables may be floating point values during some sub-problems. This can be used to develop bounds on the solution which the solver can use in proving optimality. So the objective needs to be well-defined even when the values are not integers.

Additionally, note the final values chosen by the solver may not be exactly integers either, but rather only within the solver’s allowed tolerance of an integer (e.g. `0.9999` instead of `1.0`).

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 10, 2023, 12:00pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/3 "2023-08-10T12:00:28Z")

</div>

Thanks for the reply, but this is disheartening. I have a decision problem to solve here, not really a summation problem.

I cooked up this trivial MWE to highlight the issue but in the real application I need to choose some of the columns of the matrix and process them in a non trivial way that cannot be written as just a pre-computed sum that is then multiplied by `chosen`.

In my real example the chosen columns are clustered using DBSCAN, and I want to optimize (maximize) the clustering quality. So the process is not “separatable” nor “linear”: I can’t use all columns to produce a large output and then simply “select” the ouputs.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 10, 2023, 12:25pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/4 "2023-08-10T12:25:59Z")

</div>

You could mask the original matrix instead of the sums in the same way (`m .* permutedims(chosen)`). But if you are trying to propagate JuMP VariableRef’s through an entire clustering process I don’t think that is going to work.

At this level of using JuMP (linear or quadratic programming), you are essentially formulating a problem symbolically in order to describe a problem for the solver to solve. Maybe the non-linear interface could help.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [August 10, 2023, 8:47pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/5 "2023-08-10T20:47:24Z")

</div>

> Thanks for the reply, but this is disheartening. I have a decision problem to solve here, not really a summation problem.

I hear you. I think this is a common feeling for people using JuMP for the first time who don’t come from a math programming background. It feels like JuMP is a generic optimization library, but it actually requires a very structured model in the language of mathematical programming. You can’t just thrown an arbitrary Julia function at it.

What is your `quality` function? I get that it’s not a straight summation, but here’s how I’d write your example above:

```Julia
using JuMP, HiGHS
m = rand(1000, 20)
model = Model(HiGHS.Optimizer)
@variable(model, choices[1:20], Bin)
@constraint(model, 2 <= sum(choices) <= 3)
@objective(model, Min, sum(m, dims = 1)' * choices)

```

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 11, 2023, 9:54am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/6 "2023-08-11T09:54:47Z")

</div>

Thanks for the reply guys. I have a much better understanding now.

My function cannot be written in mathematical terms. It is a clustering operation, whose purpose is to find attractors of chaotic dynamical systems. The output of the function is a real number: the clustering quality multiplied by the number of attractors found. There is no way you can get any analytic expressions there.

But I now know that it is simply not a good problem for JuMP. I will code a brute force random samling loop that samples columns, and I will also use BlackBoxOptim.jl as an alternative, as this can indeed accept any arbitrary Julia function (we have used it before already!).

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 11, 2023, 10:49am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/7 "2023-08-11T10:49:46Z")

</div>

Ah actually, BlackBoxOptim.jl is wholly unsuitable for this application: it searches in a box of real-valued variables for an optimum. I have instead a high dimensional input space, but it each dimension is binary instead of real valued.

---

<div class="post-metadata">

**Author:** ![blob](https://avatars.discourse-cdn.com/v4/letter/b/ebca7d/32.png) [@blob](https://discourse.julialang.org/u/blob)\
**Post date:** [August 11, 2023, 11:03am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/8 "2023-08-11T11:03:14Z")

</div>

If you are looking for a black-box optimization solver, maybe have a look at [NOMAD](https://github.com/bbopt/NOMAD.jl)? It allows you to use binary variables by [setting “granularity” parameter](https://bbopt.github.io/NOMAD.jl/stable/nomadProblem/#NOMAD.NomadProblem) and can take any function.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 11, 2023, 11:18am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/9 "2023-08-11T11:18:53Z")

</div>

Thanks for the suggestion, this looks very promising. I’ll try it out!

---

<div class="post-metadata">

**Author:** ![beryl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/beryl/32/209986_2.png) [@beryl](https://discourse.julialang.org/u/beryl)\
**Post date:** [October 31, 2023, 4:33pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/10 "2023-10-31T16:33:36Z")

</div>

Hi @Datseris, just wondering in the end which package worked for you? I am currently having a very similar problem in which the decision variables (integers/binaries) are used as the indices of the equations.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [October 31, 2023, 6:22pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/11 "2023-10-31T18:22:52Z")

</div>

Hi, so far I have not been able to find a solution… Let me know if you find one!

CU!

George Datseris (he/him)

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 31, 2023, 8:03pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/12 "2023-10-31T20:03:02Z")

</div>

Would it make sense to try a form of local search? That is a common approach for combinatorial optimization when mathematical programming is either infeasible or intractable.

For instance, start with a vector of `chosen` variables and then at each iteration try removing or adding every possible variable, then choose the option with the best performance.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [November 1, 2023, 8:43am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/13 "2023-11-01T08:43:25Z")

</div>

Yes this is what I do now but I call this simply the “brute force” approach, as there is no optimization, I just go through all possible combinations. This isn’t scalable but for now I am limiting myself to at most 4 max selections anyways so it works.

---

<div class="post-metadata">

**Author:** ![jd-foster](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jd-foster/32/35824_2.png) [@jd-foster](https://discourse.julialang.org/u/jd-foster)\
**Post date:** [November 1, 2023, 11:07am UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/14 "2023-11-01T11:07:07Z")

</div>

Have you considered the Metaheuristics package?

> **[GitHub - jmejia8/Metaheuristics.jl: High-performance metaheuristics for...](https://github.com/jmejia8/Metaheuristics.jl)**
>
> High-performance metaheuristics for optimization coded purely in Julia. - GitHub - jmejia8/Metaheuristics.jl: High-performance metaheuristics for optimization coded purely in Julia.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [November 1, 2023, 1:34pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/15 "2023-11-01T13:34:48Z")

</div>

I admit that from its docs it is not clear to me how I would apply this to my problem of selecting some out of a pool of variables.

CU!

George Datseris (he/him)

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 1, 2023, 3:15pm UTC](https://discourse.julialang.org/t/using-a-jump-bin-vector-to-index-an-array-checkbounds-fails-with-type-variableref/102674/16 "2023-11-01T15:15:28Z")

</div>

I had something a little more scalable in mind:

1. Start with an empty set `chosen`
2. Try adding every possible variable, and record the resulting scores
3. Add the best one
4. Go to step 2

Then you can make it more sophisticated by also removing variables