# Unstable learning of neural net with few nodes using Flux

**URL:** <https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [November 26, 2021, 2:39pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115 "2021-11-26T14:39:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Handam](https://avatars.discourse-cdn.com/v4/letter/h/ccd318/32.png) [@Handam](https://discourse.julialang.org/u/Handam)\
**Post date:** [November 26, 2021, 2:39pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/1 "2021-11-26T14:39:16Z")

</div>

I have a neural network with only one binary input node, one hidden layer and one output node. When I try to train the network, such that it outputs the value x\_1 when the input is 0 and x\_2 when the input is 1, it will sometimes work and sometimes not. I use the Flux library in Julia:

```julia
using Flux
using Flux.Optimise: update!

function train_v(v, initial_lr, target1, target2)
	for i in 1:1000
		lr = initial_lr * (1-i/1000)
		update_v(v, [0], target1, lr)
		update_v(v, [1], target2, lr)
	end
end

function update_v(v, input, target, lr)
	ps = params(v)
	gs = gradient(ps) do
		Flux.Losses.mae(v(input), target)
	end
	update!(Descent(lr), ps, gs)
end

function test(target1, target2, nodes=4, initial_lr=0.1, print = true, return_v = false)
	v = Chain(Dense(1, nodes, relu), Dense(nodes, 1))
	if print
		println("Before training: v(0):", v([0])[1], " / v(1):", v([1])[1])
		#for i in 1:3
		#	println(params(v)[i])
		#end
	end
	train_v(v, initial_lr, target1, target2)
	if print
		println("After training: v(0):", v([0])[1], " / v(1):", v([1])[1])
		#for i in 1:3
		#	println(params(v)[i])
		#end
	end
	if return_v
		return v
	end
end

```

Running for example `test(1,2)` will sometimes work just fine and result in

```julia
julia> test(1,2)
Before training: v(0):0.0 / v(1):-0.35872597
After training: v(0):0.99979866 / v(1):1.9998001

```

and sometimes not work as expected and result in outputting the same values for both inputs

```julia
julia> test(1,2)
Before training: v(0):0.0 / v(1):-0.33133784
After training: v(0):1.1958001 / v(1):1.1958001

```

Observations I made:

It is working properly more often, if…

- the absolute values of x\_1 and x\_2 are rather small.
- the difference between x\_1 and x\_2 is rather small.
- I use more nodes in the hidden layer.

Especially using a lot of nodes seem to make it work consistently. This one, however, I find especially counter intuitive, because I thought for non-complex functions it is better to use few nodes. Even with just one node in the hidden layer it is easy to find weights such that the function is exact. Can someone explain me this behaviour? The function I used for testing different settings was:

```julia
function get_success_probability(target1, target2, nodes=4, initial_lr=0.1, n=100, epsilon=abs(target1-target2)*0.1)
	success_count = 0
	for i in 1:n
		println(i)
		v = test(target1, target2, nodes, initial_lr, false, true)
		if abs(v([0])[1]-target1)<epsilon && abs(v([1])[1]-target2)<epsilon
			success_count += 1
		end
	end
	return success_count/n
end

```

I use v1.6.2.

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 26, 2021, 3:17pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/2 "2021-11-26T15:17:36Z")

</div>

Hi @Handam ,

are you sure your training loop guarantees convergence?  
Have you plotted learning curves (loss vs iteration) and compared them between runs? I think that should shed some light on what’s going on.

Also note (not sure whether that was unclear) that the weights in the `Dense` layers are initialized randomly, which is the reason for the non-determinism between runs that you observe. (You can already see that fact if you compare the “Before training” outputs of your two runs).

---

<div class="post-metadata">

**Author:** ![Handam](https://avatars.discourse-cdn.com/v4/letter/h/ccd318/32.png) [@Handam](https://discourse.julialang.org/u/Handam)\
**Post date:** [November 26, 2021, 4:09pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/3 "2021-11-26T16:09:20Z")

</div>

First of all, thanks for the answer.  
I am aware of the fact that the non-determinism comes from the random initial weights. However, I was not able to find a pattern for which initial weights it is converging and for which it is not.

I don’t know if my learning rate decay guarantees convergence, but I don’t see why it should ever result in outputting the same values for both binary inputs. Using a more “traditional” decay like \frac{\alpha}{episode} results in the same issue.

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 26, 2021, 4:13pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/4 "2021-11-26T16:13:43Z")

</div>

> [@Handam](#):
>
> I don’t see why it should ever result in outputting the same values for both binary inputs.

sorry, I missed that point (even though you explicitly mentioned it).  
However I still think you’ll probably easily understand what’s going on, once you plot things.  
(Again something you might already know, but just in case: [GitHub - JuliaLogging/TensorBoardLogger.jl: Easy peasy logging to TensorBoard with Julia](https://github.com/JuliaLogging/TensorBoardLogger.jl))

---

<div class="post-metadata">

**Author:** ![Handam](https://avatars.discourse-cdn.com/v4/letter/h/ccd318/32.png) [@Handam](https://discourse.julialang.org/u/Handam)\
**Post date:** [November 26, 2021, 4:53pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/5 "2021-11-26T16:53:31Z")

</div>

A plot when it is not working:

 ![Screenshot 2021-11-26 at 17.44.39](https://global.discourse-cdn.com/julialang/original/3X/d/8/d8a7f97ad9d7220de4b4bb54023d4fe462b4e393.png)  
So it jumps to the final value within the first training loop, which I have to admit is interesting! However, I still fail to see the reason why this is happening for some initial weights 🙈

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 26, 2021, 5:00pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/6 "2021-11-26T17:00:26Z")

</div>

Dead ReLus maybe.  
Plotting loss, gradients, histograms over weights will give you more clues.  
Try Elus instead (they should not die) and/or some regularization.

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 26, 2021, 5:05pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/7 "2021-11-26T17:05:18Z")

</div>

Just a side note:  
You’re calling `params()` and `Descent()` each iteration (actually even twice).  
While that’s not wrong it is still unnecessary (and thus inefficient)

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 26, 2021, 5:23pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/8 "2021-11-26T17:23:34Z")

</div>

> [@trahflow](#):
>
> Dead ReLus maybe.

Yep, exaclty what’s happening:

```julia
julia> v[1].weight
4×1 Matrix{Float32}:
 -0.13737935
 -0.16708376
 -0.82774603
  0.51158303

julia> v[1].bias
4-element Vector{Float32}:
  0.0
  0.0
  0.0
 -0.51735896

```

If you directly look at the weights of the first layer, you see that for `input ∈ [0, 1]`, the thing that goes into the ReLu will always be negative, thus the output of the ReLu zero, and therefore the gradient with respect to the parameters always zero.

As suggested above: A different activation function and/or regularization will help!

---

<div class="post-metadata">

**Author:** ![Handam](https://avatars.discourse-cdn.com/v4/letter/h/ccd318/32.png) [@Handam](https://discourse.julialang.org/u/Handam)\
**Post date:** [November 27, 2021, 5:52pm UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/9 "2021-11-27T17:52:38Z")

</div>

Thanks for all your answers! Helped a lot, now everything is clear to me so far! And also thanks for the hint with calling params() more often than necessary - I checked it with @btimes and it does make a difference indeed! So the best solution is to pass `ps = params(v)` as an argument to the `update_v()` function, right?

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [November 28, 2021, 10:18am UTC](https://discourse.julialang.org/t/unstable-learning-of-neural-net-with-few-nodes-using-flux/72115/10 "2021-11-28T10:18:38Z")

</div>

> [@Handam](#):
>
> So the best solution is to pass `ps = params(v)` as an argument to the `update_v()` function, right?

yes that would be reasonable