# How to use \`HyperOpt.jl\` to optimise a vector of parameters

**URL:** <https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232>\
**Category:** Machine Learning\
**Tags:** question, package\
**Created:** [June 4, 2022, 11:49am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232 "2022-06-04T11:49:53Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 4, 2022, 11:49am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/1 "2022-06-04T11:49:53Z")

</div>

I’d like to use `HyperOpt.jl` to optimise a vector of parameters, but I’m having trouble using following the libraries documentation (a readme)

As far as I can tell this should be sufficient

```julia
using Hyperopt
n = 10
d1 = rand(n)

function f(aa, d1)
    return abs.(sum(aa .- d1))
end

ho = @hyperopt for i=50,
            a = [LinRange(0,1,1000) for _ in 1:n],
    f(a,d1)
end

```

But this fails. While this works

```nohighlight
ho = @hyperopt for i=50,
            a = LinRange(0,1,1000),
    d2 = d1
    f(a,d2)
end

```

I think there are two things going on here:

1. I cannot set `a` to be a vector of `LinRange`
2. A constant has to be defined in the macro, I’m not sure why this is the case

How can I implement the first code block correctly?

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [June 6, 2022, 7:27am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/2 "2022-06-06T07:27:34Z")

</div>

Not at my computer so cant test, but i would guess that its that you call f with a linrange of length 1000, and d of length 10, so it cant broadcast them. Does it work if you make them the same length?

Edit: Also saw now that you have an extra comma after the array of linranges, that would likely mess things up. So i think both things you mention as problems should work if you just fix the comma and then maybe the dimensions, if that rurns out to be a problem.

---

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 6, 2022, 7:45am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/3 "2022-06-06T07:45:40Z")

</div>

Thanks for your reply. You are correct about the comma, that shouldn’t be there. Now I get

```nohighlight
julia> ho = @hyperopt for i=50,
                   a = [LinRange(0,1,10) for _ in 1:n]
           f(a,d1)
       end
ERROR: UndefVarError: a not defined
Stacktrace:
 [1] macro expansion
   @ ./REPL[23]:3 [inlined]
 [2] var"#26###hyperopt_objective#314"(i::Int64, state::LinRange{Float64, Int64})
   @ Main ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:151
 [3] macro expansion
   @ ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:162 [inlined]
 [4] macro expansion
   @ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938 [inlined]
 [5] optimize(ho::Hyperoptimizer{RandomSampler{Random.MersenneTwister}, var"##26###hyperopt_objective#314"})
   @ Hyperopt ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:161
 [6] top-level scope
   @ ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:193

```

This error is the same if I have `LinRange(0,1,1000)`.  
It is my understanding that the length of the `LinRange` just increases the number of discrete values which can be used for `a`.

This format also gives the same error if trying to optimise a single value of a:

```nohighlight
ho = @hyperopt for i=50,
    a = LinRange(0,1,10)
f(a,d1)
end

```

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [June 6, 2022, 7:52am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/4 "2022-06-06T07:52:32Z")

</div>

Not sure exactly what that error is complaining about. Maybe something messed up in you session, try restart the repl and make sure the function can be called with what you expect manually and try printing what it gets from the optimization loop just to check it aligns.

What would be the reason for optimizing over a vector of identical linranges btw? Seems like there will be very little optimization? I assume it might just have been an example, but wanted to check so there wasnt some misunderstanding in how it worked.

---

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 6, 2022, 8:08am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/5 "2022-06-06T08:08:49Z")

</div>

Unfortunately a fresh session doesn’t fix it. I’m not sure why it says `a` is not defined. Calling `f(1,d1)` and `f(ones(n),d1)` returns what I’d expect. I cannot print from the optimisation loop as it says `a` is not defined.

For my actual use case they won’t all have identical bounds, it’d be more like `[LinRange(0,1,50), LinRange(80,90,50),LinRange(115,125,50),...]`. As they are in this example we are optimising 10 distinct coefficients, they just happen to have the same bounds.

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [June 6, 2022, 8:19am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/6 "2022-06-06T08:19:21Z")

</div>

Seems like there is a problem with internally generated function overwriting each other when there is only one parameter to optimize. A simple workaround would be to just define another dummy variable that doesn’t need to be used like

```julia
julia> ho = @hyperopt for i=50,
                   a = [LinRange(0,1,10) for _ in 1:n],
                   dummy = 1
           f(a,d1)
       end

```

though it should probably be fixed in some nicer way, or at least a warning in the readme about this problem.

[https://github.com/baggepinnen/Hyperopt.jl/issues/69](https://github.com/baggepinnen/Hyperopt.jl/issues/69)

---

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 6, 2022, 9:32am UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/7 "2022-06-06T09:32:44Z")

</div>

Thanks, this now runs but I’m not really sure what it is doing the correct thing. Here this chosen value of `a` is a `LinRange`

```julia
ho = @hyperopt for i=50,
                          a = [LinRange(0,1,10) for _ in 1:n],
                          dummy = 1
                  f(a,d1)
              end
Hyperoptimizer with
  1 length: 10
  2 length: 1
  minimum / maximum: (2.9106243780834062, 2.9106243780834062)
  minimizer:
        a dummy 
LinRange{Float64}(0.0, 1.0, 10) 1 

```

I can optimise a vector of parameters using this method.

```nohighlight
using Hyperopt
using Optim: optimize, Options, minimum, minimizer

function f(aa, d1)
    return sum(abs.(aa .- d1))
end

objective = function (resources::Real, pars::AbstractVector)
    res = optimize(x->f(x, d1), pars, Options(time_limit=resources/100))
    minimum(res), minimizer(res)
end

candidates = [LinRange(0,1,5) for _ in 1:n] # A vector of vectors also works, but parameters will not get nice names in plots
ho = hyperband(objective, candidates; R=5, η=3, threads=true)

```

But I preferred the macro syntax.

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [June 6, 2022, 1:28pm UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/8 "2022-06-06T13:28:25Z")

</div>

I think those two are doing different things, the first is optimizing over a single variable where the options are n identical linranges, and the other one is optimizing over n variables, where each variable has the possible values in the linrange.

So if you want to express the second one using the macro syntax I would guess it is something similar to this

```julia
ho = @hyperopt for i = 50,
        v1 = LinRange(0,1,5),
        v2 = LinRange(0,1,5),
        v3 = LinRange(0,1,5),
        v4 = LinRange(0,1,5),
        v5 = LinRange(0,1,5),
        v6 = LinRange(0,1,5),
        v7 = LinRange(0,1,5),
        v8 = LinRange(0,1,5),
        v9 = LinRange(0,1,5),
        v10 = LinRange(0,1,5)
    f([v1, v2, v3, v4, v5, v6, v7, v8, v9, v10], d1)
end

```

which seems to run for me. Though if you want to use hyperband I haven’t used that and don’t know more that there is in the [readme](https://github.com/baggepinnen/Hyperopt.jl#hyperband), so maybe try something like that?

---

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 6, 2022, 2:17pm UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/9 "2022-06-06T14:17:37Z")

</div>

Writing our each variable is what I’d like to avoid,  
my vector could be quite large and it seems silly to write out each argument. Also I don’t see how this

```julia
ho = @hyperopt for i = 50,
        v1 = LinRange(0,1,5),
        v2 = LinRange(0,1,5),
        v3 = LinRange(0,1,5),
        v4 = LinRange(0,1,5),
        v5 = LinRange(0,1,5),
        v6 = LinRange(0,1,5),
        v7 = LinRange(0,1,5),
        v8 = LinRange(0,1,5),
        v9 = LinRange(0,1,5),
        v10 = LinRange(0,1,5)
    f([v1, v2, v3, v4, v5, v6, v7, v8, v9, v10], d1)
end

```

is different to this?

```julia
ho = @hyperopt for i = 50,
        v = [LinRange(0,1,5) for _ in 1:10]
    f(v1,d1)
end

```

Unless the `hyperopt` macro, as you suggest the `hyperopt` is unable to see each variable in a vector as independent and in need of optimising.

I think I’ll give up on the `hyperopt` macro, and just use hyperband. Thank you for your help.

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [June 6, 2022, 2:37pm UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/10 "2022-06-06T14:37:33Z")

</div>

The hyperopt macro will look at each variable as a vector of somethings, and sample from the values in that vector as the variable. Look for example at

```julia-repl
julia> ho = @hyperopt for i = 50,
               a = [LinRange(1, 2, 10) for _ in 1:5], dummy=1
           @show a
       end
a = range(1.0, stop=2.0, length=10)
...

```

where you see that a is not a 5 long vector with values sampled from the 5 individual linranges, but it picks one of the 5 linranges as the variable.

It might be possible to do what you want with the macro, but then I think you would need to generate the product of all parameter spaces you have or something like this

```julia
ho = @hyperopt for i = 50,
        a = collect(Iterators.product((LinRange(1, 2, 10) for _ in 1:5)...)), 
        dummy=1
    @show a
end

```

which is doable if it is more convenient for you, but collecting that full parameter space might be large if you have a lot of parameters with a lot of possible values, so depending on that it might be simpler to just use another interface which can sample from the individual ones without the need of generating the full space.

---

<div class="post-metadata">

**Author:** ![this\_josh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/this_josh/32/42679_2.png) [@this\_josh](https://discourse.julialang.org/u/this_josh)\
**Post date:** [June 6, 2022, 3:26pm UTC](https://discourse.julialang.org/t/how-to-use-hyperopt-jl-to-optimise-a-vector-of-parameters/82232/11 "2022-06-06T15:26:25Z")

</div>

I think I need to improve my understanding of macros, I didn’t expect it to just pick a variable within the vector. As you suggest, I think I’ll use an alternative interface.