# Parameters not updating in Flux

**URL:** <https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [July 7, 2021, 12:21pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208 "2021-07-07T12:21:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 12:21pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/1 "2021-07-07T12:21:33Z")

</div>

I have the following custom layer.

```julia
struct Activation{F}
    f::F
    Activation(f::Function) = new{typeof(f)}(f)
end
(m::Activation)(x::AbstractArray) = m.f.(x)

```

I test whether it allows a model to be trained using the following code

```julia
using Flux

function test_training(model,x,y)
    opt = Descent(0.1)
    loss = Flux.Losses.mse
    losses = Vector{Float32}(undef,2)
    for i = 1:2
        local loss_val
        ps = Flux.Params(Flux.params(model))
        gs = gradient(ps) do
            predicted = model(x)
            loss_val = loss(predicted,y)
        end
        losses[i] = loss_val
        Flux.Optimise.update!(opt,ps,gs)
    end
    if losses[1]==losses[2]
        error("Parameters not updating.")
    end
    return nothing
end

x = ones(Float32,4,4,1,1)
y = ones(Float32,4,4,2,1)
model = Chain(Conv((3, 3), 1=>2,pad=SamePad()),Activation(tanh))
test_training(model,x,y)

```

The model does get trained on my machine (Windows), but for some reasons fails on all operating systems when tested in Github Actions.

The following type unstable variant of the custom layer does work in Github Actions.

```julia
struct Activation
    f::Function
end
(m::Activation)(x::AbstractArray) = m.f.(x)

```

I checked Julia and Flux versions on Github Actions and they are the same as on my machine.

Does anyone have any ideas on what is happening here? A comment on whether it works on your machine or not also helps.

EDIT:

Changing `opt = Descent(0.1)` to `opt = ADAM()` allows checks to pass. However, still do not know why it fails when using `Descent`.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 7, 2021, 12:48pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/2 "2021-07-07T12:48:23Z")

</div>

Have you checked if ps contains your parameters? I think you are missing the registration of your model with Flux.functor?

See this for example  
functor(::Type{\<:Chain}, c) = c.layers, ls → Chain(ls…)

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 7, 2021, 12:51pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/3 "2021-07-07T12:51:57Z")

</div>

This is the link i wanted to copy paste

> <https://github.com/FluxML/Flux.jl/blob/9410677b3bbb94ef5af95dccf71aa9867e1c5e46/src/layers/basic.jl#L143>

Sorry, i am on ipad.

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 12:54pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/4 "2021-07-07T12:54:19Z")

</div>

I did test with `@functor` even though parameters were successfully obtained on my machine, but it did not resolve the issue.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 7, 2021, 4:40pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/5 "2021-07-07T16:40:12Z")

</div>

What exactly is the failure you’re running into here? Is it an error? If so, can you post a full stacktrace + MWE with `@functor`? If not, can you explain in detail what the failure mode on GH actions looks like?

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 6:29pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/6 "2021-07-07T18:29:31Z")

</div>

Error condition in the `test_training` (function in the first post) gets triggered. Meaning that my custom layer somehow prevented training of a model.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 7, 2021, 7:27pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/7 "2021-07-07T19:27:15Z")

</div>

Sorry that I cannot help more, I am on vacation till monday away from Real computer.

There are few things I do not understand. Your example is not complete, as there is no model defined. I wonder why you define inner constructor in the first place. You do not need to do that to have type stable model. Can you add the definition of the model?

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 7:42pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/8 "2021-07-07T19:42:21Z")

</div>

It is okay, you are trying to help and I appreciate it 🙂

The model is defined in the second field with code. You need to scroll down to see it.

> [@a-ill](#):
>
> `model = Chain(Conv((3, 3), 1=>2,pad=SamePad()),Activation(tanh))`

I use such a struct because it allows me to define activation layers as `Activation(f::Function)` instead of `x -> f(x)`.

If I do not use the inner construct, then the resulting layer is not type stable.

```julia
julia> @code_warntype Activation(tanh)(x)
Variables
  m::Activation
  x::Array{Float32, 4}

Body::Any
1 ─ %1 = Base.getproperty(m, :f)::Function
│ %2 = Base.broadcasted(%1, x)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{4}, Nothing, _A, Tuple{Array{Float32, 4}}} where _A
│ %3 = Base.materialize(%2)::Any
└── return %3

```

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 7, 2021, 7:48pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/9 "2021-07-07T19:48:03Z")

</div>

This doesn’t help with understanding what’s going on, but why not simply use `tanh` as the activation function of the `Conv` layer?

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 7:49pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/10 "2021-07-07T19:49:44Z")

</div>

I need separate activation layers for my project.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 7, 2021, 7:54pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/11 "2021-07-07T19:54:08Z")

</div>

When it comes to troubleshooting, I would look at the following:

1. What are the gradients for each of the params? If they’re nothing or zero, then we should look at the AD side of things.
2. Relatedly, when `losses[1] != losses[2]` locally, what is the actual difference? It may be that a nondeterministic version (e.g. something threaded or from XNNPack) is being picked up locally because of available artifacts, CPU feature detection, # of threads, etc. but not on the CI machine.
3. Does the behaviour persist with a larger learning rate? A 3x3 conv that maps 1 =\> 2 channels does not have a lot of parameters, so this should be easy to test.

Edit: I just tested locally and got no error.

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 7:59pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/12 "2021-07-07T19:59:12Z")

</div>

I am trying to define a custom type stable layer that will allow me to define activations as Activation(f::Function). I tested whether my custom layer worked with Flux by running `test_training(model,x,y)` function, which runs the model with my layer for two iterations and checks whether it trains. The model did train on my machine, but failed to do that on Github Action triggering the error `error("Parameters not updating.")` in my testing function. I am now trying to understand why it happened. After some tinkering I found that type unstable variant of the `Activation` struct does not trigger the error. Changing `opt` from `Descent` to `ADAM` also allows the model to be trained.

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 7, 2021, 8:03pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/13 "2021-07-07T20:03:28Z")

</div>

I am comparing losses because if the model parameters got updated, then the losses must be different. It is easier than to compare parameters directly.

I am going to try to use more channels to see if this makes a difference as welll as increase the learning rate.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 7, 2021, 8:06pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/14 "2021-07-07T20:06:33Z")

</div>

Sure, but it won’t help you root cause this. Please follow the points above and see what outputs you get from Github Actions. If you aren’t already doing so, I’d also run `versioninfo()` and `Pkg.status()` as part of the CI run to double check you’re on the right Julia and package versions. If possible, link the runs here so we can look at the full output too.

---

<div class="post-metadata">

**Author:** ![dhairyagandhi96](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dhairyagandhi96/32/7589_2.png) [@dhairyagandhi96](https://discourse.julialang.org/u/dhairyagandhi96)\
**Post date:** [July 8, 2021, 8:58am UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/15 "2021-07-08T08:58:25Z")

</div>

Any reason for the activation layer to not be a function? It seems like if things are pure functions (without parameters) then there are no parameters to update, and a regular function would do the trick as well

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [July 8, 2021, 10:30am UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/16 "2021-07-08T10:30:03Z")

</div>

Might not be the problem you are seeing, but in the MWE the Activation only takes arrays, but flux layers always broadcast the activation function so when used in a flux layer it will only see scalars.

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 8, 2021, 1:07pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/17 "2021-07-08T13:07:16Z")

</div>

Since I have some other layers tested in the same file as `Activation` I tried removing them to see whether it would make a difference. In that case, for some reason, the model parameters do get updated. Here are the links to the succesful CI run and `runtests.jl` file: [CI](https://github.com/OML-NPA/FluxExtra.jl/actions/runs/1011687541), [file](https://github.com/OML-NPA/FluxExtra.jl/blob/fc66d5a8c7dd5a4a41dfdd2d94014f50dcdb579d/test/runtests.jl); and the failed one: [CI](https://github.com/OML-NPA/FluxExtra.jl/actions/runs/1011644944), [file](https://github.com/OML-NPA/FluxExtra.jl/blob/ec9adf770d2b64aee69d6998da90de65f10b0d85/test/runtests.jl).

I am not sure what to think of the result. Quirks of Github Actions?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 8, 2021, 11:27pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/18 "2021-07-08T23:27:56Z")

</div>

Based on the CI output and order of operations in the second file, it seems like `test_training` is failing on [this line](https://github.com/OML-NPA/FluxExtra.jl/blob/ec9adf770d2b64aee69d6998da90de65f10b0d85/test/runtests.jl#L48) and not the `Activation` test. That would also explain why moving the `Activation` test to another file helped: it was throwing an error before it even got there!

As a side note, I’d recommend looking through [Unit Testing · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Test/) for how to structure test suites and use Julia’s built-in testing functionality. That would’ve helped you figure out which test set was failing instead of having to hunt through the entire file.

---

<div class="post-metadata">

**Author:** ![a-ill](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/a-ill/32/20663_2.png) [@a-ill](https://discourse.julialang.org/u/a-ill)\
**Post date:** [July 9, 2021, 8:12am UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/19 "2021-07-09T08:12:26Z")

</div>

Thanks for the link! `@testset` macro will be very useful here.

Could you tell me how did you end up with the line 48? Based on the stack trace, as I see it, the other line is to blame:

`[4] top-level scope @ ~/work/FluxExtra.jl/FluxExtra.jl/test/runtests.jl:142`

which tests the activation layer:

`[3] test(model::Chain{Tuple{Conv{2, 4, typeof(identity), Array{Float32, 4}, Vector{Float32}}, Activation{typeof(tanh)}}}, x::Array{Float32, 4}, y::Array{Float32, 4}) @ Main ~/work/FluxExtra.jl/FluxExtra.jl/test/runtests.jl:35`

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 9, 2021, 2:41pm UTC](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208/20 "2021-07-09T14:41:49Z")

</div>

My mistake, I thought the `@info` logging was unconditional for some reason. In that case, my working theory would be that all the updates from the previous calls to `test_training` leave the weights of `test_layer` on some kind of local minima or saddle point by the time it reaches the activation test. That could also explain why using Adam fixes the problem, as the additional momentum and adaptive update are meant to help with escaping from such points.

My recommendation is to re-initialize (i.e. re-declare) `test_layer` and all the other mutable state currently shared between tests for _every_ test. It’s very cheap to do so and will help prevent tests from interfering with one another like may be happening here.

[Next page](https://discourse.julialang.org/t/parameters-not-updating-in-flux/64208.md?page=2)