# Why the Loss function does not decrease significantly in Flux.jl

**URL:** <https://discourse.julialang.org/t/why-the-loss-function-does-not-decrease-significantly-in-flux-jl/93918>\
**Category:** Machine Learning\
**Created:** [February 2, 2023, 11:59am UTC](https://discourse.julialang.org/t/why-the-loss-function-does-not-decrease-significantly-in-flux-jl/93918 "2023-02-02T11:59:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![quantiota](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quantiota/32/46039_2.png) [@quantiota](https://discourse.julialang.org/u/quantiota)\
**Post date:** [February 2, 2023, 11:59am UTC](https://discourse.julialang.org/t/why-the-loss-function-does-not-decrease-significantly-in-flux-jl/93918/1 "2023-02-02T11:59:15Z")

</div>

After trying some optimizations on activation function and epochs value , it is not possible to fit the model to y data which is a function of the input data.

```julia
using Flux, Plots, Statistics
x = Array{Float64}(rand(5, 100));
w = [diff(x[1,:]); 0]./x[1,:];
y1 = cumsum(cos.(cumsum(w))); 
scatter(y1)
y = reshape(y1, (1, 100));
data = [(x, y)];

```

```julia
model = Chain(Dense(5 => 100), Dense(100 => 1), identity)
model[1].weight;

```

```julia
loss(m, x, y) = Flux.mse(m(x), y)
Flux.mse(model(x), y)
Flux.mse(model(x), y) == mean((model(x) .- y).^2)
opt_stat = Flux.setup(ADAM(), model)

```

```julia
loss_history = [] 

 epochs = 10000
 for epoch in 1:epochs
    
 Flux.train!(loss, model, data, opt_stat)     
        
       # print report
    train_loss = Flux.mse(model(x), y)
    push!(loss_history, train_loss)
    println("Epoch = $epoch : Training Loss = $train_loss") 
            
  end

```

```julia
ŷ = model(x)
Flux.mse(model(x), y)
Y = reshape(ŷ, (100, 1));
scatter(Y)

```

---

<div class="post-metadata">

**Author:** ![skleinbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skleinbo/32/36080_2.png) [@skleinbo](https://discourse.julialang.org/u/skleinbo)\
**Post date:** [February 2, 2023, 12:17pm UTC](https://discourse.julialang.org/t/why-the-loss-function-does-not-decrease-significantly-in-flux-jl/93918/2 "2023-02-02T12:17:39Z")

</div>

Your model is a linear function, because `Dense(in => out, σ=identity)` defaults to `identity` for its activation function. Try `model = Chain(Dense(5 => 100, relu), Dense(100 => 1, relu))` for instance. That gives an excellent fit.

The `identity` in the last layer of the `Chain` doesn’t do anything by the way.

---

<div class="post-metadata">

**Author:** ![quantiota](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quantiota/32/46039_2.png) [@quantiota](https://discourse.julialang.org/u/quantiota)\
**Post date:** [February 2, 2023, 4:50pm UTC](https://discourse.julialang.org/t/why-the-loss-function-does-not-decrease-significantly-in-flux-jl/93918/3 "2023-02-02T16:50:00Z")

</div>

thanks, I assumed that the dense function would already integrate by default the sigmoid function.