# Very simple Flux model refusing to converge

**URL:** <https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129>\
**Category:** General Usage\
**Created:** [July 26, 2023, 6:48pm UTC](https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129 "2023-07-26T18:48:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jClugstor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jclugstor/32/32689_2.png) [@jClugstor](https://discourse.julialang.org/u/jClugstor)\
**Post date:** [July 26, 2023, 6:48pm UTC](https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129/1 "2023-07-26T18:48:04Z")

</div>

I’m attempting to do something very simple for now. I just want to do a nonlinear curve fitting of sine using Flux.jl. So far, the model is completely refusing to converge, and is giving some very strange results. I must be doing something wrong because I’ve done this before and it worked very well.

```julia
using Flux, Plots, Statistics

timespan = 0:0.5:4*pi
out_dat = sin.(timespan)
plot(timespan,out_dat)

hidden = 5

dat = [([x],y) for (x,y) in zip(timespan,out_dat)]

model = Flux.Chain(
        Flux.Dense(1 => hidden,relu),
        Flux.Dense(hidden => 1))	

opt_state = Flux.setup(Adam(), model)
	
loss(mod,x,y) = Flux.Losses.mse(mod(x), y)
mean([loss(model,x...) for x in dat])

meanerr = 100
i = 0
while meanerr > 0.1
    i = i+1
    Flux.train!(loss, model, dat, opt_state)
    if i%10 == 0
        println(i)
        meanerr = mean([loss(model,x...) for x in dat])
        println(meanerr)
    end
end

NNresult = vcat(model.([[t] for t in timespan])...)

plot(NNresult, seriestype = :scatter)

```

As you can see the problem is very simple. But when I try to run the code, even after literally thousands of epochs the convergence is terrible. For example after 6000 epochs I get the following output  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/8/b8395aee126268dd483bec96cdf7821fcd4c388f.png)

I’ve tried it with all kinds of different settings, activation functions, and number of hidden neurons. It’s probably something simple I’m missing, so if anyone is able to spot anything wrong I would be very grateful. Thanks

---

<div class="post-metadata">

**Author:** ![jClugstor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jclugstor/32/32689_2.png) [@jClugstor](https://discourse.julialang.org/u/jClugstor)\
**Post date:** [July 26, 2023, 8:13pm UTC](https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129/2 "2023-07-26T20:13:35Z")

</div>

It looks like I just needed to add some more layers, which surprised me, because I thought I had done it before with just one layer. Oh well.

---

<div class="post-metadata">

**Author:** ![Sevi](https://avatars.discourse-cdn.com/v4/letter/s/c67d28/32.png) [@Sevi](https://discourse.julialang.org/u/Sevi)\
**Post date:** [July 27, 2023, 7:39am UTC](https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129/3 "2023-07-27T07:39:56Z")

</div>

According to the [Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem#Arbitrary-width_case) one layer should indeed be enough to fit virtually anything – if the number of nodes in that layer is “large enough”… which I guess 5 nodes isn’t 😅

---

<div class="post-metadata">

**Author:** ![jClugstor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jclugstor/32/32689_2.png) [@jClugstor](https://discourse.julialang.org/u/jClugstor)\
**Post date:** [July 27, 2023, 12:00pm UTC](https://discourse.julialang.org/t/very-simple-flux-model-refusing-to-converge/102129/4 "2023-07-27T12:00:30Z")

</div>

I actually tried it with many different numbers of hidden neurons. Yeah, that’s why it surprised me that such a simple thing would be having such trouble converging, so I thought I was doing something wrong. Turns out that training stuff just turns out to be difficult.