# Flux training gives NaNs

**URL:** <https://discourse.julialang.org/t/flux-training-gives-nans/34424>\
**Category:** Machine Learning\
**Created:** [February 10, 2020, 4:43pm UTC](https://discourse.julialang.org/t/flux-training-gives-nans/34424 "2020-02-10T16:43:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tobydriscoll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobydriscoll/32/1843_2.png) [@tobydriscoll](https://discourse.julialang.org/u/tobydriscoll)\
**Post date:** [February 10, 2020, 4:43pm UTC](https://discourse.julialang.org/t/flux-training-gives-nans/34424/1 "2020-02-10T16:43:22Z")

</div>

I’m training a CNN on a vision problem and sometimes the model parameters become NaNs. I don’t know how to create an MWE, and it doesn’t happen every time.

The model itself is

```julia
	Chain(
		# input 96x96x1
		Conv((5,5), 1=>32, pad=(2,2), relu), # now 96x96x32
		MaxPool((3,3),pad=(1,1),stride=2), # now 48x48x32

		Conv((5,5), 32=>32, pad=(2,2), relu),
		MeanPool((3,3),pad=(1,1),stride=2), # now 24x24x32

		Conv((5,5), 32=>64, pad=(2,2), relu),
		MeanPool((3,3),pad=(1,1),stride=2), # now 12x12x64

		Conv((5,5), 64=>64, pad=(0,0), relu), # now 8x8x64 
		MeanPool((2,2)), # now 4x4x64

		Conv((4,4), 64=>128, pad=(0,0), relu), # now 1x1x128 

		# Reshape tensor before dense layer
		x -> reshape(x, :, size(x, 4)),
		Dense(128, 5),

		softmax
	)

```

I’m using ADAM on cross-entropy loss. This in Julia 1.3.1 and Flux 0.10.1.

It runs on a CPU with about 70K gray images of size 96x96. Again, often the loss decreases fine, but then it will kick to a NaN in the weights of the first layer. Any ideas on what could be happening?

---

<div class="post-metadata">

**Author:** ![tobydriscoll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobydriscoll/32/1843_2.png) [@tobydriscoll](https://discourse.julialang.org/u/tobydriscoll)\
**Post date:** [February 11, 2020, 2:10pm UTC](https://discourse.julialang.org/t/flux-training-gives-nans/34424/2 "2020-02-11T14:10:45Z")

</div>

To answer myself, the problem was that `crossentropy` does not guard against taking the log of zero. So when a softmax result underflows to zero in the ground truth category, the result is NaN and you are SOL.

My fix is to broadcast a `max` with machine epsilon of the model result before calling on `crossentropy`.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [February 14, 2020, 8:11pm UTC](https://discourse.julialang.org/t/flux-training-gives-nans/34424/3 "2020-02-14T20:11:56Z")

</div>

You can also use `logitcrossentropy`:

```julia
julia> Flux.crossentropy([0], [0])
NaN

julia> Flux.logitcrossentropy([0], [0])
-0.0

```

---

<div class="post-metadata">

**Author:** ![cirobr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cirobr/32/219994_2.png) [@cirobr](https://discourse.julialang.org/u/cirobr)\
**Post date:** [July 3, 2023, 6:40pm UTC](https://discourse.julialang.org/t/flux-training-gives-nans/34424/4 "2023-07-03T18:40:51Z")

</div>

I am running on almost same issue here, are you still using this fix? May I please ask you to elaborate on how you have implemented it?

I have a CNN that produces no NaN’s, with Flux.logitbinarycrossentropy() which, by the way, has no **eps** parameter as argument.

When the loss function is changed to, say, Flux.focal\_loss() or Flux.dice\_coeff(), then the NaN’s plague shows up. As opposite to the above loss function, both of them do have a default value for the **eps** argument. For instance:

> focal\_loss(ŷ, y; dims=1, agg=mean, gamma=2, eps=eps(eltype(ŷ)))

Anyway, have tried to play with its value, from 10e-7 to 1e0 at the actual model training procedure, no success. For instance:

> julia\> Flux.focal\_loss([0], [0])  
> 0.0
> 
> julia\> Flux.focal\_loss([0], [0], eps=1e-7)  
> 0.0
> 
> julia\> Flux.focal\_loss([0], [0], eps=1e-7) == 1e-7  
> false
> 
> julia\> Flux.focal\_loss([0], [0], eps=1e-7) == 0  
> true

Any help is appreciated. Thanks.