# Fix parameter when passing to optimizer

**URL:** <https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102>\
**Category:** General Usage\
**Tags:** data\_structures\
**Created:** [May 13, 2021, 6:56pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102 "2021-05-13T18:56:23Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 13, 2021, 6:56pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/1 "2021-05-13T18:56:23Z")

</div>

When doing the optimization step, I often want to fix some parameters of the model.  
With parameters being a `NamedTuple`, or `ComponentArray`, my current solution is to split to organized two sets of parameters (free and fixed). Only the `free` one is changed by the optimizer, while the union of two is passed to the model evaluation. It is not great since the {free}+{fixed} is an object of a different structure than {free} itself. I think there must be a better way.

Any experience with similar problems?

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 13, 2021, 6:56pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/2 "2021-05-13T18:56:58Z")

</div>

@jonniedie could you please give some thoughts?

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [May 13, 2021, 8:26pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/3 "2021-05-13T20:26:04Z")

</div>

One solution is

> **[GitHub - rafaqz/FieldMetadata.jl: Metadata for julia fields](https://github.com/rafaqz/FieldMetadata.jl)**
>
> Metadata for julia fields. Contribute to rafaqz/FieldMetadata.jl development by creating an account on GitHub.

together with Flatten.jl

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [May 13, 2021, 8:31pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/4 "2021-05-13T20:31:05Z")

</div>

To be honest, I’d like a nicer way to do this. Simulink has a cool feature where you can tag parameters in your model as “tuneable” and it will automatically include them as optimization variables. What’s really cool about it is you don’t have to declare which variables are optimization variables anywhere in your model (or struct definition for our case). So you can easily switch around with which variables you’d like to optimize over. I’d like to prototype something similar soon. I think lenses might end up being useful.

But my typical approach with ComponentArrays is to have a function that gets called in your optimization loop to update the chosen parameters before being passed into the model. So it looks something like:

```julia
# All parameters for the model
p = ComponentArray(a=1.0, b=[5.5, 2.3], c=(a=2, b=[0, 0, 0]))

# Function to update the free parameters
function update!(p, new_p)
    p.a = new_p.a
    p.c.b .= new_p.c.b
end

# Free parameters for optimization
free_p = ComponentArray(a=p.a, c=(b=p.c.b))

```

Then `free_p` will be the parameters that get passed into your optimization routine to update the full parameters `p` in your model at each step of the optimization.

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [May 13, 2021, 8:39pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/5 "2021-05-13T20:39:48Z")

</div>

With that said, I think this is one of the exciting ideas of ModelingToolkit’s ability to use symbolic variables for both simulation and optimization. So this would be a lot easier to do there once all of the infrastructure is in place.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 13, 2021, 8:47pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/6 "2021-05-13T20:47:17Z")

</div>

nice to hear that a similar problem appears elsewhere.  
Indeed, I want to define tunable parameters and also have a possibility to shift some parameters from one category to other, `fixparameter`, `releaseparameter`. Here is [my non-elegant solution](https://github.com/mmikhasenko/AlgebraPDF.jl/blob/master/src/parameters.jl#L66) for recording.

Looking forward to your prototype.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 13, 2021, 8:50pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/7 "2021-05-13T20:50:29Z")

</div>

Thanks a lot, @hendri54  
That is super interesting and useful. Is there a way to add the Metadata to other things rather than structures?

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 13, 2021, 8:53pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/8 "2021-05-13T20:53:12Z")

</div>

@jonniedie how can symbolic variable help with fixing parameters?

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [May 13, 2021, 9:23pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/9 "2021-05-13T21:23:24Z")

</div>

[Here’s an example of optimization using ModelingToolkit](https://mtk.sciml.ai/stable/tutorials/optimization/), although I don’t think there needs to be a distinction between variables and parameters when defining them anymore. I think they both end up as type `Num` now and whether they are free or fixed in your problem depends on which place you pass them into the `OptimizationProblem`. This way you can shift things around from the `p` (fixed) or `u0` (free) vectors.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 14, 2021, 7:04am UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/10 "2021-05-14T07:04:54Z")

</div>

@oschulz Any ideas on the subject using `ValueShapes.jl`?

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [May 14, 2021, 9:17am UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/11 "2021-05-14T09:17:15Z")

</div>

Sure - ValueShapes has direct support for constant/non-constant parameters:

```julia
using ValueShapes, Parameters, LinearAlgebra, Optim

vs = NamedTupleShape(
    a = ScalarShape{Real}(),
    b = ConstValueShape([1,2,3,4]),
    c = ArrayShape{Real}(3)
)

function f(v::NamedTuple)
    @unpack a, b, c = v
    (a-2)^2 + norm(b.-3)^2 + norm(c.-4)^2
end

# Some tests:
totalndof(vs) == 4
x_guess = zeros(totalndof(vs))
v_guess = vs(x_guess)[]
(vs >> f)(x_guess)

optresult = let vs = vs
    Optim.optimize(vs >> f, x_guess, LBFGS(), autodiff=:forward)
end

vs(Optim.minimizer(optresult))[] == (
    a = 2,
    b = [1, 2, 3, 4],
    c = [4, 4, 4]
)

```

So this is using ForwardDiff, which only supports vectors, to optimze a function defined on `NamedTuple`s, keeping parameter `b` constant (which also reduces the dimensionality of the problem, the flat real vectors `x` have length 4).

If you want random starting points, and have a prior distribution for the components of `v`, using `NamedTupleDist` makes this very natural:

```julia
using ValueShapes, Parameters, LinearAlgebra, Distributions, Optim

prior = NamedTupleDist(
    a = Normal(),
    b = ConstValueShape([1,2,3,4]),
    c = MvNormal(float(Diagonal([2,4,3])))
)

vs = varshape(prior)

function f(v::NamedTuple)
    @unpack a, b, c = v
    (a-2)^2 + norm(b.-3)^2 + norm(c.-4)^2
end

# Some tests:
totalndof(vs) == 4
x_guess = rand(unshaped(prior))
(vs >> f)(x_guess)
v_guess = rand(prior)
(f)(v_guess)

optresult = let vs = vs
    Optim.optimize(vs >> f, x_guess, LBFGS(), autodiff=:forward)
end

```

I plan to replace `vs >> f` by `unshaped(f, vs)`, it’ll be clearer.

BAT.jl is able to do variate transformations using such priors, so that the optimizer can run in an infinite space even if parameters have bounds (as specified by their prior distributions). I’m planning to spin that off as a separate package sometime soonish.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 14, 2021, 9:24am UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/12 "2021-05-14T09:24:52Z")

</div>

Fantastic!  
the `Contant` property is immutable, right?

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [May 14, 2021, 10:00am UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/13 "2021-05-14T10:00:10Z")

</div>

> [@misha\_mikhasenko](#):
>
> the `Contant` property is immutable

`ConstValueShape` just wraps whatever you stuff inside of it. Semantically, it’s immutable, but it can’t prevent you from modifying the array inside if you get your hands on it, of course.

---

<div class="post-metadata">

**Author:** ![ptoche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ptoche/32/23554_2.png) [@ptoche](https://discourse.julialang.org/u/ptoche)\
**Post date:** [May 14, 2021, 10:06am UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/14 "2021-05-14T10:06:16Z")

</div>

I’m pretty sure what you’re looking for appears somewhere [here](https://github.com/FRBNY-DSGE/DSGE.jl), but the architecture is so rich in features that it was overkill for my own projects.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 14, 2021, 1:56pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/15 "2021-05-14T13:56:28Z")

</div>

I mean, either the parameter is a `constant` or not is a part of the type specification, `NamedTuple{(..., ConstantShape, ...)}`. The property `constant = fasle|true` is immutable, right?

In principle it is good for the compiler. But you cannot fix and release parameters without changing the type, it seems

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [May 14, 2021, 2:17pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/16 "2021-05-14T14:17:55Z")

</div>

Not that I am aware of.

There are really two ways of setting up a model.

The one that I prefer is to build a model out of objects that keep track of their own parameters (including bounds, what is fixed, etc). This allows me to compute many variations of a model without having to worry about which parameters appear in each model version. For that use case, something like `FieldMetaData.jl` is helpful. (I use something else that I cooked up myself, but a similar idea)

The other approach is to keep track of the relevant model parameters (and of where they belong in the model) “by hand.” This is what is discussed in the rest of this thread. Then `FieldMetaData.jl` is probably not a good solution.

A “flattened” implementation (where parameters don’t live in the individual model objects) is contained in  
[https://frbny-dsge.github.io/ModelConstructors.jl/stable/](https://frbny-dsge.github.io/ModelConstructors.jl/stable/)  
which is part of the Fed’s DSGE package that @ptoche pointed to. It has the capability of switching individual parameters between fixed and estimated.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [May 14, 2021, 2:55pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/17 "2021-05-14T14:55:12Z")

</div>

> [@misha\_mikhasenko](#):
>
> I mean, either the parameter is a `constant` or not is a part of the type specification

Oh, yes it’s part of the type, so the compiler can optimize for it, but changing which parameters are constant requires new code generation.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 15, 2021, 7:28pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/18 "2021-05-15T19:28:28Z")

</div>

The next thing I try will be a paramete structure with `ComponentArray`, and `StaticVector` for `fixed`/`free` status. Somthing like this

```julia
struct FliggedNamedTuple{T}
   parameters::ComponentArray{T}
   flags::BitVector
end

```

I should be able to change values and the status without mutating.

---

<div class="post-metadata">

**Author:** ![jonniedie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonniedie/32/12842_2.png) [@jonniedie](https://discourse.julialang.org/u/jonniedie)\
**Post date:** [May 15, 2021, 8:02pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/19 "2021-05-15T20:02:54Z")

</div>

In that case you might as well make the `flags` field a `ComponentArray{Bool}` with the same structure as your parameters so you can set whether things are fixed or free by name:

```julia
struct FlaggedNamedTuple{T,N,A1,A2,Ax}
   parameters::ComponentArray{T,N,A1,Ax}
   flags::ComponentArray{Bool,N,A2,Ax}
end

```

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [May 15, 2021, 8:06pm UTC](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102/20 "2021-05-15T20:06:28Z")

</div>

the dimension of flats is `length(keys(parameters))`, I thought I do not want to fix subcomponents.  
Although, why not. Maybe I do.  
Thanks for the excellent idea!

[Next page](https://discourse.julialang.org/t/fix-parameter-when-passing-to-optimizer/61102.md?page=2)