Optimisation design advice - metaprogramming/closures

Hello. We are trying to optimise parameters for functions that look something like:

function cost_func(time::Vector{Float64}, params::Vector{Float64})
    cₐ, a, cᵦ, β = params
    return (some complicated combination of time and parameters)

The number of parameters varies for different models but we always extract them explicitly from the array for stylistic reasons and to avoid typos in the model definition.

What is the best way to ‘freeze’ one or more of the parameters to a specific value(s) so that the optimisation is only performed on the ‘unfrozen’ parameters?

Note that the freezing must be able to take place within another function so we must avoid world age issues. Also, we must be optimisation algorithm/library generic so we can’t just set lower and upper limits to just below and above the desired value for the fixed parameters (as some algo/libraries don’t feature bounds).

At the moment we have a rather complex solution defining our models as expressions and using functionwrappers to avoid world age issues. I suspect that we can do something far more straightforward with closures but I haven’t found a way to do it yet.

Many thanks for any advice or thoughts.

1 Like

Closures? But it seems you already know about them, so I am not sure what the question is.

If I understand the questions correctly, the usual way to do something like this with closures is

f(a, b, c) = a + b + c

optimize((b, c) -> f(5, b, c))

or similar. This way you only have to provide two parameters and a is set as 5.

1 Like

Hello @Tamas_Papp and @Karajan , thanks both for your responses.

Yes I agree that would be a nice way to do it, but how is it possible when the arguments takes the form of a vector of floats? Is there a zero-overhead way to add some abstraction on top of that? And also, it needs to be done in a generic, programatic way (the programmer doesn’t know which parameters the user will want to freeze).

(edit: in case it’s worth mentioning, a model in our software is actually a struct with four different possible functions that can be fitted/predicted against depending the available data, all use the same parameters. The struct can also contain additional info such as params as a tuple of symbols)

I’m not sure I understand correctly what the specific setting is for you but I’ll try.

You can do the following:

# I'm using a buffer array here so I don't have to allocate a new
# array when going from length 3 to 4.
function wrapper(optim_array, buffer_array, index, value)
    found = 0
    for i in 1:length(buffer_array)
        if i == index
            buffer_array[i] = value
            found = 1
            buffer_array[i] = optim_array[i - found]
    @show buffer_array

buffer_array = zeros(4)
optimize(x -> wrapper(x, buffer_array, 3, 5), ...)

It hasn’t zero overhead but pretty close to it.

cost_function(x) = sum(x)

@btime wrapper($(ones(3)), $(zeros(4)), 3, 5)
  11.918 ns (0 allocations: 0 bytes)

There are various simple ways to do this using higher-order functions, but please keep in mind that asking for a fully optimized, zero-overhead, and convenient solution without providing an MWE is somewhat unrealistic.

@Karajan many thanks for your suggested solution. Although I didn’t end up using it, the idea of a wrapper function was exactly what I needed so I am very grateful. In case it is of use to anyone at all I present a minimum working example of the way I eventually designed the software. Note that in benchmarks there is no significant overhead.

The main components:

struct ModelStruct

function ModelStruct(;name::String="Custom model", free::Tuple=(), fixed::NamedTuple=NamedTuple{}(), G)
   ModelStruct(name, free, fixed, G)

function freeze_params(orig::ModelStruct, tofix::NamedTuple)
    newfree =  Tuple([i for i in orig.free_params if !(i in keys(tofix))])
    newfixed = merge(orig.fixed_params, tofix)
    ModelStruct(orig.name, newfree, newfixed, orig.G)

function moduluswitharrayinput(t_or_ω::Vector{Float64}, numparams::Vector{Float64}, modulusfunc::Function, freeparams::Tuple, fixedargs::NamedTuple)
    freeargs = NamedTuple{freeparams}(numparams)
    args = merge(freeargs, fixedargs)
    modulusfunc.(t_or_ω; args...)

And how this looks in use:

function test_function1(t; η, kᵦ, kᵧ)
    kᵧ + kᵦ*exp(-t*kᵦ/η)

tester = ModelStruct(name="Test model", free = (:η, :kᵦ, :kᵧ), G = test_function1)
freeze_params(tester, (η = 1.0,))

So the ModelStruct keeps track of which parameters are free and which are fixed. The optimiser only ‘sees’ a wrapper function moduluswitharrayinput which combines the fixed parameters with the ones it is trying to optimise for and sends these combined to the actual objective function.