# Combining Lux.jl with IPNewton() for Optimization

**URL:** <https://discourse.julialang.org/t/combining-lux-jl-with-ipnewton-for-optimization/114391>\
**Category:** General Usage\
**Tags:** question, lux\
**Created:** [May 17, 2024, 10:01am UTC](https://discourse.julialang.org/t/combining-lux-jl-with-ipnewton-for-optimization/114391 "2024-05-17T10:01:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![paul188](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paul188/32/207567_2.png) [@paul188](https://discourse.julialang.org/u/paul188)\
**Post date:** [May 17, 2024, 10:01am UTC](https://discourse.julialang.org/t/combining-lux-jl-with-ipnewton-for-optimization/114391/1 "2024-05-17T10:01:14Z")

</div>

I have some specific bounds on my Neural Network weights and biases, that cannot be expressed as simple parameter inequlality constraints. That’s why I would like to try something like IPNewton for the Optimization, even though it is slower and possibly not converging.

The problem now is that IPNewton accepts arrays for the optimization, whereas Lux.jl accepts ComponentVectors of a special form. Specifically, to convert an array of weights and biases in the Optimization routine to a ComponentVector is difficult. I wrote my own function for this. But unfortunately, I am having problems with AutoDiff and mutations.

Is there any easy, feasible way to convert arrays or Vectors to a structure that a Lux NN can accept as parameters ? This is my current function:

```julia
function convert_params_to_tuple_no_ode(p::Vector{Float64}, n_in::Int64, n_out::Int64, hidden_layers::Tuple{Int64, Int64})
    # Create the layers tuple: input layer, hidden layers, output layer
    layers = (n_in, fill(hidden_layers[2], hidden_layers[1])..., n_out)
    
    final_idx = 1

    nn_subtuple = NamedTuple()

    for layer_nr in 1:(length(layers)-1)
        weight_size = layers[layer_nr] * layers[layer_nr + 1]
        bias_size = layers[layer_nr + 1]
        
        # Extract weights and biases from the parameter vector
        weights = p[final_idx:(final_idx + weight_size - 1)]
        final_idx += weight_size
        weights = reshape(weights, layers[layer_nr + 1], layers[layer_nr])
        
        biases = p[final_idx:(final_idx + bias_size - 1)]
        final_idx += bias_size
        
        # Create a vector for biases
        bias_vec = biases # Bias is usually a vector, not a matrix
        
        # Create a named tuple for the current layer
        subtuple_layer = (weight = weights, bias = bias_vec)
        layer_symbol = Symbol("layer_", layer_nr)
        # Create a temporary tuple for the current layer and merge it into nn_subtuple
        temp_tuple = (layer_symbol => subtuple_layer,)
        nn_subtuple = (;nn_subtuple, temp_tuple)
    end

    return ComponentArrays.ComponentVector((ps_lux = nn_subtuple,))
end

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 17, 2024, 10:49am UTC](https://discourse.julialang.org/t/combining-lux-jl-with-ipnewton-for-optimization/114391/2 "2024-05-17T10:49:24Z")

</div>

> [@paul188](#):
>
> Specifically, to convert an array of weights and biases in the Optimization routine to a ComponentVector is difficult.

It’s one line, `convert(CType,p)` where `CType` is the `typeof(ComponentArray(nn_p))` which can just be computed at the Lux.setup outside of the loss function so `CType` can just be a constant. It’s literally one line, delete all of this.

---

<div class="post-metadata">

**Author:** ![paul188](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paul188/32/207567_2.png) [@paul188](https://discourse.julialang.org/u/paul188)\
**Post date:** [May 19, 2024, 2:52pm UTC](https://discourse.julialang.org/t/combining-lux-jl-with-ipnewton-for-optimization/114391/3 "2024-05-19T14:52:14Z")

</div>

Oh my god thank you. I didn’t see the forest for the trees.