Optimisation design advice - metaprogramming/closures

General Usage
, ,
1

Hello. We are trying to optimise parameters for functions that look something like:

function cost_func(time::Vector{Float64}, params::Vector{Float64})
    cₐ, a, cᵦ, β = params
    return (some complicated combination of time and parameters)
end

The number of parameters varies for different models but we always extract them explicitly from the array for stylistic reasons and to avoid typos in the model definition.

What is the best way to ‘freeze’ one or more of the parameters to a specific value(s) so that the optimisation is only performed on the ‘unfrozen’ parameters?

Note that the freezing must be able to take place within another function so we must avoid world age issues. Also, we must be optimisation algorithm/library generic so we can’t just set lower and upper limits to just below and above the desired value for the fixed parameters (as some algo/libraries don’t feature bounds).

At the moment we have a rather complex solution defining our models as expressions and using functionwrappers to avoid world age issues. I suspect that we can do something far more straightforward with closures but I haven’t found a way to do it yet.

Many thanks for any advice or thoughts.

1 Like
2

Closures? But it seems you already know about them, so I am not sure what the question is.

3

If I understand the questions correctly, the usual way to do something like this with closures is

f(a, b, c) = a + b + c

optimize((b, c) -> f(5, b, c))

or similar. This way you only have to provide two parameters and a is set as 5.

1 Like
4

Hello @Tamas_Papp and @Karajan , thanks both for your responses.

Yes I agree that would be a nice way to do it, but how is it possible when the arguments takes the form of a vector of floats? Is there a zero-overhead way to add some abstraction on top of that? And also, it needs to be done in a generic, programatic way (the programmer doesn’t know which parameters the user will want to freeze).

(edit: in case it’s worth mentioning, a model in our software is actually a struct with four different possible functions that can be fitted/predicted against depending the available data, all use the same parameters. The struct can also contain additional info such as params as a tuple of symbols)

5

I’m not sure I understand correctly what the specific setting is for you but I’ll try.

You can do the following:

# I'm using a buffer array here so I don't have to allocate a new
# array when going from length 3 to 4.
function wrapper(optim_array, buffer_array, index, value)
    found = 0
    for i in 1:length(buffer_array)
        if i == index
            buffer_array[i] = value
            found = 1
        else
            buffer_array[i] = optim_array[i - found]
        end
    end
    @show buffer_array
    cost_function(buffer_array)
end

buffer_array = zeros(4)
optimize(x -> wrapper(x, buffer_array, 3, 5), ...)

It hasn’t zero overhead but pretty close to it.

cost_function(x) = sum(x)

@btime wrapper($(ones(3)), $(zeros(4)), 3, 5)
  11.918 ns (0 allocations: 0 bytes)
8.0
2 Likes
6

There are various simple ways to do this using higher-order functions, but please keep in mind that asking for a fully optimized, zero-overhead, and convenient solution without providing an MWE is somewhat unrealistic.

7

@Karajan many thanks for your suggested solution. Although I didn’t end up using it, the idea of a wrapper function was exactly what I needed so I am very grateful. In case it is of use to anyone at all I present a minimum working example of the way I eventually designed the software. Note that in benchmarks there is no significant overhead.

The main components:

struct ModelStruct
    name::String
    free_params::Tuple
    fixed_params::NamedTuple
    G::Function
end

function ModelStruct(;name::String="Custom model", free::Tuple=(), fixed::NamedTuple=NamedTuple{}(), G)
   ModelStruct(name, free, fixed, G)
end

function freeze_params(orig::ModelStruct, tofix::NamedTuple)
    newfree =  Tuple([i for i in orig.free_params if !(i in keys(tofix))])
    newfixed = merge(orig.fixed_params, tofix)
    ModelStruct(orig.name, newfree, newfixed, orig.G)
end

function moduluswitharrayinput(t_or_ω::Vector{Float64}, numparams::Vector{Float64}, modulusfunc::Function, freeparams::Tuple, fixedargs::NamedTuple)
    freeargs = NamedTuple{freeparams}(numparams)
    args = merge(freeargs, fixedargs)
    modulusfunc.(t_or_ω; args...)
end

And how this looks in use:

function test_function1(t; η, kᵦ, kᵧ)
    kᵧ + kᵦ*exp(-t*kᵦ/η)
end

tester = ModelStruct(name="Test model", free = (:η, :kᵦ, :kᵧ), G = test_function1)
freeze_params(tester, (η = 1.0,))

So the ModelStruct keeps track of which parameters are free and which are fixed. The optimiser only ‘sees’ a wrapper function moduluswitharrayinput which combines the fixed parameters with the ones it is trying to optimise for and sends these combined to the actual objective function.