# Flux.jl is not modifying the loss

**URL:** <https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646>\
**Category:** Machine Learning\
**Created:** [June 25, 2019, 11:07am UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646 "2019-06-25T11:07:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![francesco.alemanno](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@francesco.alemanno](https://discourse.julialang.org/u/francesco.alemanno)\
**Post date:** [June 25, 2019, 11:08am UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646/1 "2019-06-25T11:08:00Z")

</div>

```julia
using Flux,LinearAlgebra
function MWE()
    η=0.01
    model=Conv((64, 64), 1=>1, relu,stride=30)
    loss(x,y) = Flux.mse(model(x),y)
    dataset=[(rand(100,100,1,1),rand(2,2,1,1))]
    @show model(dataset[1][1])
    @show loss(dataset[1]...)
    for i in 1:10
        Flux.train!(loss,params(model), dataset, ADAM(η))
        @show i, loss(dataset[1]...)
    end
end

MWE();

```

```
model((dataset[1])[1]) = [0.0 2.86116; 0.0 0.0] (tracked)
loss(dataset[1]...) = 1.4128901344841402 (tracked)
(i, loss(dataset[1]...)) = (1, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (2, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (3, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (4, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (5, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (6, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (7, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (8, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (9, 0.37165153102154674 (tracked))
(i, loss(dataset[1]...)) = (10, 0.37165153102154674 (tracked))

```

```julia

```

For some weird reason i’m not able to fix this behavior, is there a remedy?

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [June 25, 2019, 11:18am UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646/2 "2019-06-25T11:18:31Z")

</div>

Why do you allocate a new ADAM optimizer every loop iteration? Try pulling this out of the loop (so you allocate once), although I doubt that that’s the root of the issue. Everything looks correct to me otherwise.

---

<div class="post-metadata">

**Author:** ![francesco.alemanno](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@francesco.alemanno](https://discourse.julialang.org/u/francesco.alemanno)\
**Post date:** [June 25, 2019, 2:31pm UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646/3 "2019-06-25T14:31:59Z")

</div>

> [@francesco.alemanno](#):
>
> using Flux,LinearAlgebra function MWE() η=0.01 model=Conv((64, 64), 1=\>1, relu,stride=30) loss(x,y) = Flux.mse(model(x),y) dataset=[(rand(100,100,1,1),rand(2,2,1,1))] @show model(dataset[1][1]) @show loss(dataset[1]…) for i in 1:10 Flux.train!(loss,params(model), dataset, ADAM(η)) @show i, loss(dataset[1]…) end end MWE();

same outcome

```julia
using Flux,LinearAlgebra
function MWE()
    η=0.01
    model=Conv((64, 64), 1=>1, relu,stride=30)
    loss(x,y) = Flux.mse(model(x),y)
    dataset=[(rand(100,100,1,1),rand(2,2,1,1))]
    @show model(dataset[1][1])
    @show loss(dataset[1]...)
    opt=ADAM(η)
    for i in 1:10
        Flux.train!(loss,params(model), dataset, opt)
        @show i, loss(dataset[1]...)
    end
end

MWE();

```

```
model((dataset[1])[1]) = [0.0 0.0; 1.46359 0.0] (tracked)
loss(dataset[1]...) = 0.2001447729236289 (tracked)
(i, loss(dataset[1]...)) = (1, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (2, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (3, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (4, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (5, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (6, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (7, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (8, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (9, 0.29602579662457984 (tracked))
(i, loss(dataset[1]...)) = (10, 0.29602579662457984 (tracked))

```

```julia

```

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [June 25, 2019, 2:56pm UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646/4 "2019-06-25T14:56:28Z")

</div>

I think this is simply because you’re using `relu` at the final output with only a single layer — if the model’s output ever goes negative then Flux completely loses the trail since the derivatives at that point also become zero.

---

<div class="post-metadata">

**Author:** ![francesco.alemanno](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@francesco.alemanno](https://discourse.julialang.org/u/francesco.alemanno)\
**Post date:** [June 25, 2019, 3:05pm UTC](https://discourse.julialang.org/t/flux-jl-is-not-modifying-the-loss/25646/5 "2019-06-25T15:05:46Z")

</div>

Thank you! unfortunately i was familiar with the meaning of RELU, i thought it was a linear correspondence!  
now it works