# How to apply Transfer Learning with Flux

**URL:** <https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250>\
**Category:** Machine Learning\
**Created:** [December 4, 2020, 1:24pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250 "2020-12-04T13:24:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mariana](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mariana/32/15064_2.png) [@Mariana](https://discourse.julialang.org/u/Mariana)\
**Post date:** [December 4, 2020, 1:24pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/1 "2020-12-04T13:24:07Z")

</div>

I have a trained neural network, which is already giving good results, and I’d like to use the parameters previously obtained for the model as the starting parameters to train the ANN on a new task (basically, it’s a transfer learning problem). So far, I’m training the network by using the following function"

```julia
function flux_training(x_train::Array{Float64,2}, y_train::Array{Float64,2}, n_epochs::Int, lambda::Int)
    model = Chain(Dense(54,54,sigmoid),Dense(54,54,sigmoid),Dense(54,12,leakyrelu))
    loss(x,y) = Flux.mse(model(x),y) 
    ps = params(model)
    dataset = Flux.Data.DataLoader(x_train', y_train', batchsize = 32, shuffle = true)
    opt = Optimiser(WeightDecay(lambda), ADAGrad())
    evalcb() = @show(loss(x_train', y_train'))
    for epoch in 1:n_epochs
        println("Epoch $epoch")
        time = @elapsed Flux.train!(loss, ps, dataset, opt, cb=throttle(evalcb,3))
    end
    
    y_hat = model(x_train')' 

    return y_hat, model

end

```

and I save the model created by doing:

```julia
weights = params(model)
using BSON: @save
@save "mymodel.bson" weights

```

How can I initialize the weights in my training function as the values that were previously saved, to train the ANN for a new task?

---

<div class="post-metadata">

**Author:** ![andrewdinhobl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrewdinhobl/32/23614_2.png) [@andrewdinhobl](https://discourse.julialang.org/u/andrewdinhobl)\
**Post date:** [December 4, 2020, 4:01pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/2 "2020-12-04T16:01:32Z")

</div>

I’m not sure how up-to-date it is, but there is an [example](https://www.github.com/FluxML/model-zoo/tree/master/tutorials%2Ftransfer_learning) of transfer learning in the Flux model zoo.

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [December 4, 2020, 5:32pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/3 "2020-12-04T17:32:34Z")

</div>

That example shows a nice way to freeze part of the weights and train the rest, if you just want to reload and retrain all the weights the [documentation](https://fluxml.ai/Flux.jl/stable/saving/#Saving-Model-Weights-1) has an example.

The model weights are initialized to random values ([usually](https://fluxml.ai/Flux.jl/stable/utilities/#Layer-Initialization-1), you can modify that too if you need) when the layers are constructed. If you then load new values or otherwise modify them, that will be the starting point for training.

---

<div class="post-metadata">

**Author:** ![Mariana](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mariana/32/15064_2.png) [@Mariana](https://discourse.julialang.org/u/Mariana)\
**Post date:** [December 4, 2020, 6:03pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/4 "2020-12-04T18:03:59Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![Mariana](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mariana/32/15064_2.png) [@Mariana](https://discourse.julialang.org/u/Mariana)\
**Post date:** [December 4, 2020, 6:08pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/5 "2020-12-04T18:08:15Z")

</div>

Thank you, @contradict!! This idea of initializing the weights with a pre-defined value, instead of random values, is a nice possibility, but I’m not sure how to do that…

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [December 4, 2020, 6:23pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/6 "2020-12-04T18:23:02Z")

</div>

Flux has a function `loadparams!` which replaces params of an existing model. Its a bit clunky to use as you need to keep the code which creates the model structure around.

I think BSON can save the whole Chain so you dont need to do this (i.e. do `BSON.@save model` instead).

You could also try [ONNXmutable.jl](https://github.com/DrChainsaw/ONNXmutable.jl) for more long term storage.

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [December 4, 2020, 10:49pm UTC](https://discourse.julialang.org/t/how-to-apply-transfer-learning-with-flux/51250/7 "2020-12-04T22:49:51Z")

</div>

> [@Mariana](#):
>
> initializing the weights with a pre-defined value

It the initializers are not particularly useful for that since you have to specify them at layer creation time. This facility is mostly useful for experimenting with new layer types or perhaps scaling the random initialization to fit some peculiarity of your specific problem. `loadparams!` or one of the other methods @DrChainsaw mentioned is probably the correct solution to your problem.