# Flux.train! hangs up after updating packages

**URL:** <https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772>\
**Category:** New to Julia\
**Tags:** flux, ode, neural-network\
**Created:** [June 29, 2021, 5:04pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772 "2021-06-29T17:04:35Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex0](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex0/32/21169_2.png) [@Alex0](https://discourse.julialang.org/u/Alex0)\
**Post date:** [June 29, 2021, 5:04pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/1 "2021-06-29T17:04:35Z")

</div>

Hello people,  
I am currently trying to train a neural ODE network using a chemistry network from Catalyst.jl.  
My [code](https://github.com/aspecht1/neuralODE/blob/main/test.jl) was working fine until I updated my packages. Unfortunatley I just did `Pkg.update()` and didnt really pay attention to what was updated, so I cant really go back.

What is happening now, is that `Flux.train!` seems to get stuck right away (no error, it just keeps running forever). Before the update it took me 90sec for 100 training iterations (on a very weak machine).

I am using Atom/Juno to write my code and this is showing up:  
 ![inf](https://global.discourse-cdn.com/julialang/original/3X/3/2/3220367d9b1efbd628c76162b3bae00b1d30c160.png)  
Maybe someone with Atom/Juno experience knows what this means in my case?

I am using Julia 1.61, DiffEqFlux 1.41, Flux 0.12.4, DiffEqSensivity 6.54.0 and DifferentialEquations 6.17.1.  
Its hard to figure out what wrong without error messages … my code was working as it is before the updates.  
Any ideas whats wrong/happening?

EDIT: maybe I should mention that I batched up my code from the [documentation](https://diffeqflux.sciml.ai/stable/examples/neural_ode_flux/) and this [paper](https://arxiv.org/abs/2103.15341v2).

---

<div class="post-metadata">

**Author:** ![aramirezreyes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aramirezreyes/32/42573_2.png) [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Post date:** [June 29, 2021, 5:15pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/2 "2021-06-29T17:15:03Z")

</div>

Not sure this is the case, but did you restart Julia after updating the packages?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [June 29, 2021, 5:48pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/3 "2021-06-29T17:48:09Z")

</div>

Try obtaining gradients on just one batch of data and not using `train!`. That should give you a fast way to troubleshoot any issues.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 29, 2021, 6:25pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/4 "2021-06-29T18:25:27Z")

</div>

Yes, show the gradient calculation example as an MWE.

---

<div class="post-metadata">

**Author:** ![Alex0](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex0/32/21169_2.png) [@Alex0](https://discourse.julialang.org/u/Alex0)\
**Post date:** [June 30, 2021, 1:14pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/5 "2021-06-30T13:14:37Z")

</div>

Okay so I updated my code like [this](https://github.com/aspecht1/neuralODE/blob/main/test.jl) to look at the gradients and using `update!`, but now the training somehow works fine. I am happy with that but this doesnt really allow me to understand why `train!` is not working for me. `update!` is also showing much better results than `train!` (when it was working).

I guess this must be because of my model because the example code from the [documentation](https://diffeqflux.sciml.ai/stable/examples/neural_ode_flux/) using `train!` works just fine. But its still strange that it happened after updating my packages.  
So I dont know how to make an MWE out of my code/model or whether it makes sense.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 30, 2021, 2:45pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/6 "2021-06-30T14:45:25Z")

</div>

You’re restructuring outside of the function:

[https://github.com/aspecht1/neuralODE/blob/main/test.jl#L34](https://github.com/aspecht1/neuralODE/blob/main/test.jl#L34)

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [June 30, 2021, 4:11pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/7 "2021-06-30T16:11:45Z")

</div>

Also, `ForwardDiff.gradient` is different from `Zygote.gradient`, which Flux uses under the hood. You may be able to repro the issue by using the latter instead.

---

<div class="post-metadata">

**Author:** ![Alex0](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex0/32/21169_2.png) [@Alex0](https://discourse.julialang.org/u/Alex0)\
**Post date:** [July 2, 2021, 7:08pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/8 "2021-07-02T19:08:48Z")

</div>

Okay, sorry for the late response. So I replaced `grad = ForwardDiff.gradient()` with `@time grad = Zygote.gradient()[1]` and got this output:

> 1.0966585823313262e7  
> 177.862870 seconds (287.27 M allocations: 20.921 GiB, 5.49% gc time, 32.76% compilation time)  
> ┌ Warning: First function call produced NaNs. Exiting.  
> └ @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/2AoGt/src/initdt.jl:81  
> ┌ Warning: Automatic dt set the starting dt as NaN, causing instability.  
> └ @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/2AoGt/src/solve.jl:510  
> ┌ Warning: NaN dt detected. Likely a NaN value in the state, parameters, or derivative value caused this outcome.  
> └ @ SciMLBase ~/.julia/packages/SciMLBase/kCcpg/src/integrator\_interface.jl:325  
> 445024.28042782197  
> ┌ Warning: First function call produced NaNs. Exiting.  
> └ @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/2AoGt/src/initdt.jl:81  
> ┌ Warning: Automatic dt set the starting dt as NaN, causing instability.  
> └ @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/2AoGt/src/solve.jl:510  
> ┌ Warning: NaN dt detected. Likely a NaN value in the state, parameters, or derivative value caused this outcome.  
> └ @ SciMLBase ~/.julia/packages/SciMLBase/kCcpg/src/integrator\_interface.jl:325

Then the code hangs up again. So like you assumed the problem is calculating the gradient (with Zygote). `Zygote.gradient()` needs 177s to calculate the gradient, while `ForwardDiff.gradient()` only needs 0.02s.

---

<div class="post-metadata">

**Author:** ![Alex0](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex0/32/21169_2.png) [@Alex0](https://discourse.julialang.org/u/Alex0)\
**Post date:** [July 2, 2021, 7:51pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/9 "2021-07-02T19:51:33Z")

</div>

> [@ChrisRackauckas](#):
>
> You’re restructuring outside of the function:

Is that a problem and which function are talking about? Im very new to this, so Im not quite sure what you are pointing to. But I am very happy to learn/get more insight how to do these things the best way.  
So all the example codes I looked at are doing it in a similar way.  
With `p, re = Flux.destructure(NN)` I obtain the initial network parameters and the `re` function to reconstruct the network, that I name `re_NN` in the line your referencing.

I will look deeper into it.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 2, 2021, 7:57pm UTC](https://discourse.julialang.org/t/flux-train-hangs-up-after-updating-packages/63772/10 "2021-07-02T19:57:23Z")

</div>

> [@Alex0](#):
>
> Is that a problem and which function are talking about? Im very new to this, so Im not quite sure what you are pointing to. But I am very happy to learn/get more insight how to do these things the best way.  
> So all the example codes I looked at are doing it in a similar way.  
> With `p, re = Flux.destructure(NN)` I obtain the initial network parameters and the `re` function to reconstruct the network, that I name `re_NN` in the line your referencing.

If you don’t restructure in the ODE, the neural network is not a function of the parameters `p`.

```julia
re_NN = re(p)

function dudt!(du, u, p, t) #scale the data
    du .= re(p)(u) .* yscale / t_end
end

```