Fix parameter when passing to optimizer

General Usage
1

When doing the optimization step, I often want to fix some parameters of the model.
With parameters being a NamedTuple, or ComponentArray, my current solution is to split to organized two sets of parameters (free and fixed). Only the free one is changed by the optimizer, while the union of two is passed to the model evaluation. It is not great since the {free}+{fixed} is an object of a different structure than {free} itself. I think there must be a better way.

Any experience with similar problems?

1 Like
2

@jonniedie could you please give some thoughts?

3

One solution is

together with Flatten.jl

1 Like
4

To be honest, I’d like a nicer way to do this. Simulink has a cool feature where you can tag parameters in your model as “tuneable” and it will automatically include them as optimization variables. What’s really cool about it is you don’t have to declare which variables are optimization variables anywhere in your model (or struct definition for our case). So you can easily switch around with which variables you’d like to optimize over. I’d like to prototype something similar soon. I think lenses might end up being useful.

But my typical approach with ComponentArrays is to have a function that gets called in your optimization loop to update the chosen parameters before being passed into the model. So it looks something like:

# All parameters for the model
p = ComponentArray(a=1.0, b=[5.5, 2.3], c=(a=2, b=[0, 0, 0]))

# Function to update the free parameters
function update!(p, new_p)
    p.a = new_p.a
    p.c.b .= new_p.c.b
end

# Free parameters for optimization
free_p = ComponentArray(a=p.a, c=(b=p.c.b))

Then free_p will be the parameters that get passed into your optimization routine to update the full parameters p in your model at each step of the optimization.

1 Like
5

With that said, I think this is one of the exciting ideas of ModelingToolkit’s ability to use symbolic variables for both simulation and optimization. So this would be a lot easier to do there once all of the infrastructure is in place.

6

nice to hear that a similar problem appears elsewhere.
Indeed, I want to define tunable parameters and also have a possibility to shift some parameters from one category to other, fixparameter, releaseparameter. Here is my non-elegant solution for recording.

Looking forward to your prototype.

7

Thanks a lot, @hendri54
That is super interesting and useful. Is there a way to add the Metadata to other things rather than structures?

8

@jonniedie how can symbolic variable help with fixing parameters?

9

Here’s an example of optimization using ModelingToolkit, although I don’t think there needs to be a distinction between variables and parameters when defining them anymore. I think they both end up as type Num now and whether they are free or fixed in your problem depends on which place you pass them into the OptimizationProblem. This way you can shift things around from the p (fixed) or u0 (free) vectors.

1 Like
10

@oschulz Any ideas on the subject using ValueShapes.jl?

11

Sure - ValueShapes has direct support for constant/non-constant parameters:

using ValueShapes, Parameters, LinearAlgebra, Optim

vs = NamedTupleShape(
    a = ScalarShape{Real}(),
    b = ConstValueShape([1,2,3,4]),
    c = ArrayShape{Real}(3)
)

function f(v::NamedTuple)
    @unpack a, b, c = v
    (a-2)^2 + norm(b.-3)^2 + norm(c.-4)^2
end

# Some tests:
totalndof(vs) == 4
x_guess = zeros(totalndof(vs))
v_guess = vs(x_guess)[]
(vs >> f)(x_guess)

optresult = let vs = vs
    Optim.optimize(vs >> f, x_guess, LBFGS(), autodiff=:forward)
end

vs(Optim.minimizer(optresult))[] == (
    a = 2,
    b = [1, 2, 3, 4],
    c = [4, 4, 4]
)

So this is using ForwardDiff, which only supports vectors, to optimze a function defined on NamedTuples, keeping parameter b constant (which also reduces the dimensionality of the problem, the flat real vectors x have length 4).

If you want random starting points, and have a prior distribution for the components of v, using NamedTupleDist makes this very natural:

using ValueShapes, Parameters, LinearAlgebra, Distributions, Optim

prior = NamedTupleDist(
    a = Normal(),
    b = ConstValueShape([1,2,3,4]),
    c = MvNormal(float(Diagonal([2,4,3])))
)

vs = varshape(prior)

function f(v::NamedTuple)
    @unpack a, b, c = v
    (a-2)^2 + norm(b.-3)^2 + norm(c.-4)^2
end

# Some tests:
totalndof(vs) == 4
x_guess = rand(unshaped(prior))
(vs >> f)(x_guess)
v_guess = rand(prior)
(f)(v_guess)

optresult = let vs = vs
    Optim.optimize(vs >> f, x_guess, LBFGS(), autodiff=:forward)
end

I plan to replace vs >> f by unshaped(f, vs), it’ll be clearer.

BAT.jl is able to do variate transformations using such priors, so that the optimizer can run in an infinite space even if parameters have bounds (as specified by their prior distributions). I’m planning to spin that off as a separate package sometime soonish.

1 Like
12

Fantastic!
the Contant property is immutable, right?

13

ConstValueShape just wraps whatever you stuff inside of it. Semantically, it’s immutable, but it can’t prevent you from modifying the array inside if you get your hands on it, of course.

14

I’m pretty sure what you’re looking for appears somewhere here, but the architecture is so rich in features that it was overkill for my own projects.

15

I mean, either the parameter is a constant or not is a part of the type specification, NamedTuple{(..., ConstantShape, ...)}. The property constant = fasle|true is immutable, right?

In principle it is good for the compiler. But you cannot fix and release parameters without changing the type, it seems

16

Not that I am aware of.

There are really two ways of setting up a model.

The one that I prefer is to build a model out of objects that keep track of their own parameters (including bounds, what is fixed, etc). This allows me to compute many variations of a model without having to worry about which parameters appear in each model version. For that use case, something like FieldMetaData.jl is helpful. (I use something else that I cooked up myself, but a similar idea)

The other approach is to keep track of the relevant model parameters (and of where they belong in the model) “by hand.” This is what is discussed in the rest of this thread. Then FieldMetaData.jl is probably not a good solution.

A “flattened” implementation (where parameters don’t live in the individual model objects) is contained in
https://frbny-dsge.github.io/ModelConstructors.jl/stable/
which is part of the Fed’s DSGE package that @ptoche pointed to. It has the capability of switching individual parameters between fixed and estimated.

1 Like
17

Oh, yes it’s part of the type, so the compiler can optimize for it, but changing which parameters are constant requires new code generation.

1 Like
18

The next thing I try will be a paramete structure with ComponentArray, and StaticVector for fixed/free status. Somthing like this

struct FliggedNamedTuple{T}
   parameters::ComponentArray{T}
   flags::BitVector
end

I should be able to change values and the status without mutating.

19

In that case you might as well make the flags field a ComponentArray{Bool} with the same structure as your parameters so you can set whether things are fixed or free by name:

struct FlaggedNamedTuple{T,N,A1,A2,Ax}
   parameters::ComponentArray{T,N,A1,Ax}
   flags::ComponentArray{Bool,N,A2,Ax}
end
1 Like
20

the dimension of flats is length(keys(parameters)), I thought I do not want to fix subcomponents.
Although, why not. Maybe I do.
Thanks for the excellent idea!