How to use `HyperOpt.jl` to optimise a vector of parameters

[this_josh]

I’d like to use HyperOpt.jl to optimise a vector of parameters, but I’m having trouble using following the libraries documentation (a readme)

As far as I can tell this should be sufficient

using Hyperopt
n = 10
d1 = rand(n)

function f(aa, d1)
    return abs.(sum(aa .- d1))
end

ho = @hyperopt for i=50,
            a = [LinRange(0,1,1000) for _ in 1:n],
    f(a,d1)
end

But this fails. While this works

ho = @hyperopt for i=50,
            a = LinRange(0,1,1000),
    d2 = d1
    f(a,d2)
end

I think there are two things going on here:

1. I cannot set a to be a vector of LinRange
2. A constant has to be defined in the macro, I’m not sure why this is the case

How can I implement the first code block correctly?

[albheim]

Not at my computer so cant test, but i would guess that its that you call f with a linrange of length 1000, and d of length 10, so it cant broadcast them. Does it work if you make them the same length?

Edit: Also saw now that you have an extra comma after the array of linranges, that would likely mess things up. So i think both things you mention as problems should work if you just fix the comma and then maybe the dimensions, if that rurns out to be a problem.

[this_josh]

Thanks for your reply. You are correct about the comma, that shouldn’t be there. Now I get

julia> ho = @hyperopt for i=50,
                   a = [LinRange(0,1,10) for _ in 1:n]
           f(a,d1)
       end
ERROR: UndefVarError: a not defined
Stacktrace:
 [1] macro expansion
   @ ./REPL[23]:3 [inlined]
 [2] var"#26###hyperopt_objective#314"(i::Int64, state::LinRange{Float64, Int64})
   @ Main ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:151
 [3] macro expansion
   @ ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:162 [inlined]
 [4] macro expansion
   @ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938 [inlined]
 [5] optimize(ho::Hyperoptimizer{RandomSampler{Random.MersenneTwister}, var"##26###hyperopt_objective#314"})
   @ Hyperopt ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:161
 [6] top-level scope
   @ ~/.julia/packages/Hyperopt/zGNt7/src/Hyperopt.jl:193

This error is the same if I have LinRange(0,1,1000).
It is my understanding that the length of the LinRange just increases the number of discrete values which can be used for a.

This format also gives the same error if trying to optimise a single value of a:

ho = @hyperopt for i=50,
    a = LinRange(0,1,10)
f(a,d1)
end

[albheim]

Not sure exactly what that error is complaining about. Maybe something messed up in you session, try restart the repl and make sure the function can be called with what you expect manually and try printing what it gets from the optimization loop just to check it aligns.

What would be the reason for optimizing over a vector of identical linranges btw? Seems like there will be very little optimization? I assume it might just have been an example, but wanted to check so there wasnt some misunderstanding in how it worked.

[this_josh]

Unfortunately a fresh session doesn’t fix it. I’m not sure why it says a is not defined. Calling f(1,d1) and f(ones(n),d1) returns what I’d expect. I cannot print from the optimisation loop as it says a is not defined.

For my actual use case they won’t all have identical bounds, it’d be more like [LinRange(0,1,50), LinRange(80,90,50),LinRange(115,125,50),...]. As they are in this example we are optimising 10 distinct coefficients, they just happen to have the same bounds.

[albheim]

Seems like there is a problem with internally generated function overwriting each other when there is only one parameter to optimize. A simple workaround would be to just define another dummy variable that doesn’t need to be used like

julia> ho = @hyperopt for i=50,
                   a = [LinRange(0,1,10) for _ in 1:n],
                   dummy = 1
           f(a,d1)
       end

though it should probably be fixed in some nicer way, or at least a warning in the readme about this problem.

https://github.com/baggepinnen/Hyperopt.jl/issues/69

[this_josh]

Thanks, this now runs but I’m not really sure what it is doing the correct thing. Here this chosen value of a is a LinRange

ho = @hyperopt for i=50,
                          a = [LinRange(0,1,10) for _ in 1:n],
                          dummy = 1
                  f(a,d1)
              end
Hyperoptimizer with
  1 length: 10
  2 length: 1
  minimum / maximum: (2.9106243780834062, 2.9106243780834062)
  minimizer:
        a     dummy 
LinRange{Float64}(0.0, 1.0, 10)         1 

I can optimise a vector of parameters using this method.

using Hyperopt
using Optim: optimize, Options, minimum, minimizer

function f(aa, d1)
    return sum(abs.(aa .- d1))
end

objective = function (resources::Real, pars::AbstractVector)
    res = optimize(x->f(x, d1), pars, Options(time_limit=resources/100))
    minimum(res), minimizer(res)
end

candidates = [LinRange(0,1,5) for _ in 1:n] # A vector of vectors also works, but parameters will not get nice names in plots
ho = hyperband(objective, candidates; R=5, η=3, threads=true)

But I preferred the macro syntax.

[albheim]

I think those two are doing different things, the first is optimizing over a single variable where the options are n identical linranges, and the other one is optimizing over n variables, where each variable has the possible values in the linrange.

So if you want to express the second one using the macro syntax I would guess it is something similar to this

ho = @hyperopt for i = 50,
        v1 = LinRange(0,1,5),
        v2 = LinRange(0,1,5),
        v3 = LinRange(0,1,5),
        v4 = LinRange(0,1,5),
        v5 = LinRange(0,1,5),
        v6 = LinRange(0,1,5),
        v7 = LinRange(0,1,5),
        v8 = LinRange(0,1,5),
        v9 = LinRange(0,1,5),
        v10 = LinRange(0,1,5)
    f([v1, v2, v3, v4, v5, v6, v7, v8, v9, v10], d1)
end

which seems to run for me. Though if you want to use hyperband I haven’t used that and don’t know more that there is in the readme, so maybe try something like that?

[this_josh]

Writing our each variable is what I’d like to avoid,
my vector could be quite large and it seems silly to write out each argument. Also I don’t see how this

ho = @hyperopt for i = 50,
        v1 = LinRange(0,1,5),
        v2 = LinRange(0,1,5),
        v3 = LinRange(0,1,5),
        v4 = LinRange(0,1,5),
        v5 = LinRange(0,1,5),
        v6 = LinRange(0,1,5),
        v7 = LinRange(0,1,5),
        v8 = LinRange(0,1,5),
        v9 = LinRange(0,1,5),
        v10 = LinRange(0,1,5)
    f([v1, v2, v3, v4, v5, v6, v7, v8, v9, v10], d1)
end

is different to this?

ho = @hyperopt for i = 50,
        v = [LinRange(0,1,5) for _ in 1:10]
    f(v1,d1)
end

Unless the hyperopt macro, as you suggest the hyperopt is unable to see each variable in a vector as independent and in need of optimising.

I think I’ll give up on the hyperopt macro, and just use hyperband. Thank you for your help.

[albheim]

The hyperopt macro will look at each variable as a vector of somethings, and sample from the values in that vector as the variable. Look for example at

julia> ho = @hyperopt for i = 50,
               a = [LinRange(1, 2, 10) for _ in 1:5], dummy=1
           @show a
       end
a = range(1.0, stop=2.0, length=10)
...

where you see that a is not a 5 long vector with values sampled from the 5 individual linranges, but it picks one of the 5 linranges as the variable.

It might be possible to do what you want with the macro, but then I think you would need to generate the product of all parameter spaces you have or something like this

ho = @hyperopt for i = 50,
        a = collect(Iterators.product((LinRange(1, 2, 10) for _ in 1:5)...)), 
        dummy=1
    @show a
end

which is doable if it is more convenient for you, but collecting that full parameter space might be large if you have a lot of parameters with a lot of possible values, so depending on that it might be simpler to just use another interface which can sample from the individual ones without the need of generating the full space.

[this_josh]

I think I need to improve my understanding of macros, I didn’t expect it to just pick a variable within the vector. As you suggest, I think I’ll use an alternative interface.