# Minibatching neural ODEs with different initial conditions
**URL:** https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361
**Category:** Machine Learning
**Tags:** diffeq, diffeqflux
**Created:** [March 20, 2023, 9:10pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361 "2023-03-20T21:10:10Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@kaido975](https://discourse.julialang.org/u/kaido975)
#### Post date: [March 20, 2023, 9:10pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/1 "2023-03-20T21:10:10Z")
I am trying to train a neural ODE on different time trajectories, starting with different initial conditions. I am able to observe a reduction in loss with ADAM (with good outputs), but BFGS does not work at all. Why is that the case? What else can I try here to get better results?
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [March 21, 2023, 12:40am UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/2 "2023-03-21T00:40:17Z")
> [@kaido975](#):
>
> I am able to observe a reduction in loss with ADAM (with good outputs), but BFGS does not work at all. Why is that the case? What else can I try here to get better results?
Are you pre-starting with ADAM? BFGS can get stuck in local minima quite easily.
---
### Author:  [@kaido975](https://discourse.julialang.org/u/kaido975)
#### Post date: [March 21, 2023, 1:30am UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/3 "2023-03-21T01:30:34Z")
Yes, starting with ADAM followed by BFGS, as in the tutorials.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [March 21, 2023, 2:25am UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/4 "2023-03-21T02:25:33Z")
Show the code.
---
### Author:  [@kaido975](https://discourse.julialang.org/u/kaido975)
#### Post date: [March 21, 2023, 4:29pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/5 "2023-03-21T16:29:43Z")
Here is the code.
```julia
using Optimization, OptimizationOptimisers, OptimizationOptimJL
using Optim, Flux, Lux, OrdinaryDiffEq, ComponentArrays, Statistics
using IterTools: ncycle
using Random
ann = Lux.Chain(Lux.Dense(2,10,Lux.tanh),
Lux.Dense(10,10,Lux.tanh),
Lux.Dense(10,1))
p, st = Lux.setup(rng, ann)
group_size = 3
continuity_term = 200
function loss_function(data, pred)
return sum(abs2, data - pred)
end
function loss_multiple_shooting(p, data)
y_train = data[2:end]'
x0 = data[1]
# ODE problem parametrized by initial condition
prob = ODEProblem((u,p,t)->ann([x0, u[1]], p, st)[1], x0, tspan, p)
return multiple_shoot(p, y_train, tsteps, prob, loss_function, Tsit5(),
group_size; continuity_term)
end
#train_data contains time trajectories and initial conditions
train_loader = Flux.Data.DataLoader((train_data, ), batchsize=1)
adtype = Optimization.AutoZygote()
optf = Optimization.OptimizationFunction((x,p, batch) -> loss_multiple_shooting(x, batch), adtype)
optprob = Optimization.OptimizationProblem(optf, ComponentVector{Float32}(p))
result_neuralode = Optimization.solve(optprob, ADAM(0.01),
ncycle(train_loader, 100))
optprob2 = remake(optprob,u0 = result_neuralode.u)
result_neuralode2 = Optimization.solve(optprob2,
Optim.BFGS(initial_stepnorm =0.01),
allow_f_increases = true,
ncycle(train_loader, 20))
```
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [March 21, 2023, 5:39pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/6 "2023-03-21T17:39:41Z")
I see you’re using a data loader. BFGS doesn’t work well with stochastic loss functions
---
### Author:  [@kaido975](https://discourse.julialang.org/u/kaido975)
#### Post date: [March 21, 2023, 6:21pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/7 "2023-03-21T18:21:17Z")
Is there a way to generate multiple time series starting with different initial conditions?
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [March 21, 2023, 8:18pm UTC](https://discourse.julialang.org/t/minibatching-neural-odes-with-different-initial-conditions/96361/8 "2023-03-21T20:18:39Z")
Just solve at each. Or make it a matrix if you have an NN like that. But just make sure you do all initial conditions every time if you use BFGS. It’s not an optimizer for changing what you evaluate.