# Flux: Resample model with different initializations

**URL:** <https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [April 12, 2021, 11:53am UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097 "2021-04-12T11:53:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![v-i-s-h](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/v-i-s-h/32/46152_2.png) [@v-i-s-h](https://discourse.julialang.org/u/v-i-s-h)\
**Post date:** [April 12, 2021, 11:53am UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/1 "2021-04-12T11:53:46Z")

</div>

Hi,

I want to do an experiment in Flux where I want to keep the architecture of the model same and want to initialize the parameters with different variance normal distributions. My current set up looks like this

```julia
function build_model(var)
    init_fn(args...) = sqrt(var) * randn(args...)

    return Chain(
        Conv(...usual params..., init = init_fn),
        # Simillar layers
        Dense(...usual params..., init = init_fn)
    )
end

```

And whenever I want a model initialized to certain variance, I do

```julia
model = build_model(0.01); # model with variance = 0.01
# do some experiment
model = build_model(1.0); # model with variance = 1.0

```

My understanding is that here I am building a new network evertime I need a newly sampled model.

Is there a way where I can reuse an existing model and reinitialize the model parameters with a different set of values?

Thanks ahead,  
vish

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [April 16, 2021, 4:18pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/2 "2021-04-16T16:18:11Z")

</div>

You can access the model weights directly and re-init them in-place. It’ll take some manual work, but a couple of helper functions would alleviate that.

---

<div class="post-metadata">

**Author:** ![v-i-s-h](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/v-i-s-h/32/46152_2.png) [@v-i-s-h](https://discourse.julialang.org/u/v-i-s-h)\
**Post date:** [April 17, 2021, 2:20pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/3 "2021-04-17T14:20:44Z")

</div>

Hi @ToucheSir ,

Could please provide me any pointers/examples on how to do it?

I tried to do the following with `Dense` layer, but is getting error!

```julia
julia> m.layers[3].weight |> size
(4, 10)

> m.layers[3].weight = rand(4, 10)
ERROR: setfield! immutable struct of type Dense cannot be changed
Stacktrace:
 [1] setproperty!(x::Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, f::Symbol, v::Matrix{Float64})
   @ Base ./Base.jl:34
 [2] top-level scope
   @ REPL[11]:1

```

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [April 17, 2021, 7:43pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/4 "2021-04-17T19:43:27Z")

</div>

> [@v-i-s-h](#):
>
> `m.layers[3].weight = rand(4, 10)`

You’re trying to replace the entire parameter field of the layer (which is an immutable struct) instead of updating an array in-place. Think `m.layers[2].setWeight(rand(...))` in another language. Since the objective appears to be avoiding unnecessary allocations, you’d want to update the _contents_ of the array in place instead:

```julia
m.layers[3].weight .= rand(4, 10)

```

(note the dot)

---

<div class="post-metadata">

**Author:** ![v-i-s-h](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/v-i-s-h/32/46152_2.png) [@v-i-s-h](https://discourse.julialang.org/u/v-i-s-h)\
**Post date:** [April 21, 2021, 1:36pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/5 "2021-04-21T13:36:38Z")

</div>

Hi @ToucheSir, thanks for the pointers. I tried to follow the solution you proposed and developed a `resample_model` (Method 2 below) function for the same.

I tried to benchmark both to see which one is better. I was under the assumption that in-place update of parameters should do better. Below are the necessary snippets.

```julia-auto
# Function to build model                                                                                                                                            
function LeNet5(; initializer=Flux.glorot_uniform, act_fn=relu, im_size=(28, 28, 1), n_classes=10)                                                                   
     out_conv_size = (im_size[1] ÷ 4 - 3, im_size[2] ÷ 4 -3, 16)                                                                                                      
                                                                                                                                                                     
    return Chain(                                                                                                                                                    
        Conv((5, 5), im_size[end] => 6, act_fn, init=initializer),
        MaxPool((2, 2)),
        Conv((5, 5), 6 => 16, act_fn, init=initializer),
        MaxPool((2, 2)),
        flatten,
        Dense(prod(out_conv_size), 120, act_fn, init=initializer),
        Dense(120, 84, act_fn, init=initializer),
        Dense(84, n_classes, init=initializer)
    )                                                                                                                                                                
end                                                                                                                                                                          

```

## Method 1: Call build function everytime

```julia-auto
function test_reinit_1()
    model = LeNet5()

    for i in 1:N
        model = LeNet5()
    end

end

```

## Method 2: Inplace parameter update

```julia-auto
function resample_model!(model)
    for layer ∈ model
        if hasproperty(layer, :weight)
            layer.weight .= randn(size(layer.weight))
        end
        if hasproperty(layer, :bias)
            layer.bias .= randn(size(layer.bias))
        end
    end
end

function test_reinit_2()
    model = LeNet5()

    for i in 1:N
        resample_model!(model)
    end
end

```

## Results

Below is the benchmark results for 'N = 100`

```julia-auto
julia> @benchmark test_reinit_1()
BenchmarkTools.Trial: 
  memory estimate: 34.66 MiB
  allocs estimate: 12322
  --------------
  minimum time: 6.986 ms (0.00% GC)
  median time: 9.145 ms (14.77% GC)
  mean time: 9.143 ms (12.06% GC)
  maximum time: 12.402 ms (12.52% GC)
  --------------
  samples: 546
  evals/sample: 1

```

```julia-auto
julia> @benchmark test_reinit_2()
BenchmarkTools.Trial: 
  memory estimate: 34.86 MiB
  allocs estimate: 9522
  --------------
  minimum time: 16.408 ms (0.00% GC)
  median time: 23.207 ms (6.69% GC)
  mean time: 22.937 ms (7.06% GC)
  maximum time: 29.082 ms (9.05% GC)
  --------------
  samples: 218
  evals/sample: 1

```

It seems like method 1 (calling model building function over and over again) is better than my current implementation of method 2. This looks odd as I was expecting opposite results.

Any suggestions how I can speed up method2 here?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [April 21, 2021, 3:54pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/6 "2021-04-21T15:54:09Z")

</div>

I thought there was an in-place version of `randn`, but perhaps I was mistaken. Given the memory estimate is similar, it may be easier for you to just go with method 1.

---

<div class="post-metadata">

**Author:** ![v-i-s-h](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/v-i-s-h/32/46152_2.png) [@v-i-s-h](https://discourse.julialang.org/u/v-i-s-h)\
**Post date:** [April 22, 2021, 1:01am UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/7 "2021-04-22T01:01:54Z")

</div>

Thanks for the comments. Maybe method 1 is the best option so far.

---

<div class="post-metadata">

**Author:** ![dhairyagandhi96](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dhairyagandhi96/32/7589_2.png) [@dhairyagandhi96](https://discourse.julialang.org/u/dhairyagandhi96)\
**Post date:** [July 8, 2021, 9:07am UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/8 "2021-07-08T09:07:28Z")

</div>

The correct thing would be defining an initialisation like in the OP.

---

<div class="post-metadata">

**Author:** ![v-i-s-h](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/v-i-s-h/32/46152_2.png) [@v-i-s-h](https://discourse.julialang.org/u/v-i-s-h)\
**Post date:** [July 9, 2021, 12:14pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/9 "2021-07-09T12:14:07Z")

</div>

Hi @dhairyagandhi96 , sorry I am unable to understand what you told? How can I define an initialization? And what is OP?

---

<div class="post-metadata">

**Author:** ![dhairyagandhi96](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dhairyagandhi96/32/7589_2.png) [@dhairyagandhi96](https://discourse.julialang.org/u/dhairyagandhi96)\
**Post date:** [July 9, 2021, 12:22pm UTC](https://discourse.julialang.org/t/flux-resample-model-with-different-initializations/59097/10 "2021-07-09T12:22:38Z")

</div>

The initialisation is the `init_fn` from the original post (OP)