Fix parameter when passing to optimizer

General Usage
21

Could you help me with a constructor?

FliggedNamedTuple(v::NamedTuple) = FliggedNamedTuple(ComponentArray(v), ComponentArray(v_with_values_replaced_by_bools))

struggling to covert the type preserving the structure

EDIT:
got it

begin
	struct FliggedNamedTuple{T,Ax}
	   parameters::ComponentVector{T,Vector{T},Ax}
	   flags::ComponentVector{Bool,Vector{Bool},Ax}
	end
	# 
	function FliggedNamedTuple(v::NamedTuple)
	    cv = ComponentVector(v)
	    dof = length(cv)
	    axes = getaxes(cv)[1]
	    FliggedNamedTuple(cv, ComponentVector(fill(false,dof), axes))
	end
end
1 Like
22

The way I like doing this is by composing the objective to be optimized with a function that completes the model. I think this is pretty much what @jonniedie suggested, and is as close as it gets to how you would formulate it on paper, so it cannot be too wrong :slight_smile:

1 Like
23

@lostella, thank for the links

what are parameters in your case?
how is the composition (\circ) defined?
Is m_original.layers global?

24

Exactly, in that specific example the m_original is an object in global scope, and it contains all parameters, some of which we want to fine-tune. So get_complete_model just takes the subset of parameters we want to optimize, and puts it together with the part of m_original that we want to keep.

In my example above, the model is the composition of multiple layers where we want to fine tune only the final ones: so it takes all layers but the last three from m_original and stacks the given layers on top of these. But this could work for anything that holds your parameters, including a ComponentArray from which you only want to optimize some components, or a regular Array from which you want to optimize some coefficients, or whatever.

The principle is simple: if f is a function of all parameters, then objective = f ∘ get_complete_model is a function of the subset of parameters that you want to optimize. The composition operator \circ is straight from base Julia, so the above is equivalent to objective = x -> f(get_complete_model(x)).

1 Like
25

ok, thanks for the clarification. Nice

26

Hi all!

I just stumbled over this post since we’ve got a similar problem where we want to fix certain parameters.

The main piece of code is:

function forward_with_fixed_params(forward, params)
           forward_fixed(x) = forward((; params..., x...))
           return forward_fixed
end

Is there anything wrong with that?

julia> forward((a, b, c)) = a+b+c^C

julia> forward(x) = x.a+x.b+x.c
forward (generic function with 1 method)

julia> forward((;a=0, b=10, c=10))
20

julia> fwd_n = forward_with_fixed_params(forward, (;a=10, b=10, c=10))
(::var"#forward_fixed#1"{typeof(forward), NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}}) (generic function with 1 method)

julia> fwd_n((;a=100))
120

julia> @code_warntype fwd_n((;a=100))
MethodInstance for (::var"#forward_fixed#1"{typeof(forward), NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}})(::NamedTuple{(:a,), Tuple{Int64}})
  from (::var"#forward_fixed#1")(x) in Main at REPL[1]:2
Arguments
  #self#::var"#forward_fixed#1"{typeof(forward), NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}}
  x::NamedTuple{(:a,), Tuple{Int64}}
Body::Int64
1 ─ %1 = Core.getfield(#self#, :forward)::Core.Const(forward)
│   %2 = Base.NamedTuple()::Core.Const(NamedTuple())
│   %3 = Core.getfield(#self#, :params)::NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}
│   %4 = Base.merge(%2, %3)::NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}
│   %5 = Base.merge(%4, x)::NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}
│   %6 = (%1)(%5)::Int64
└──      return %6


julia> @time fwd_n((;a=100))
  0.000004 seconds
120
1 Like