Hyperopt.jl Baysean Optimization Hyperband for Neural Networks

Tooling
1

Has anyone successfully found optimal neural networks using Hyperopt.jl’s Baysean Optimization Hyperband (BOHB) ?

I seem to have run into the fundamental issue that from what I can tell, that the package wants the state returned to be a Union{Tuple, Vector} of the sampled parameters. From the source code:

# struct to record BOHB Observation
mutable struct ObservationsRecord
    dim::Int
    observation::Union{Vector, Tuple}
    loss::Real
end
function ObservationsRecord(observation, loss)
    ObservationsRecord(length(observation), observation, loss)
end

But as a state, and I want to return a full neural network model to resume training on, with all the weights and biases. This will never be my sample space. My sample space will be e.g. the number of neurons.

In my particular case, I am saving neural networks in a custom struct:

Base.@kwdef mutable struct NeuralNetwork
    ID
    model::Chain
    loss_function::Function
    optimizer::Flux.Optimise.AbstractOptimiser
    trained_epochs = 0
    trained_time = 0.0
    current_loss_train = NaN
    current_performance = NaN
    history = (trained_epochs=Int64[], trained_time=Float64[], loss_train=Float64[], performance=Float64[], wallclock=Float64[])
    last_written = eval(:(time()))  # Evaluates at construction-time
end

Returning this object is fine when the inner sample is RandomSampler(), and I can use it to resume training no problem. But BOHB is not pleased, as it can not determine the length of a NeuralNetwork, nor can it convert it to Union{Tuple, Vector}.

Do anyone know how to properly set up BOHB optimization of neural networks?