Is there a way to use optimization algorithms from Optim to train Lux neural networks and neural ODEs? In particular, I would like to use LBFGS to train a Lux neural network but don’t see a simple way to do this. There is a library called FluxOptimTools that allows us to do this in Flux.
See Neural Ordinary Differential Equations · DiffEqFlux.jl. You can essentially use the Optimization.jl API to train Lux using Optim (and several other optimization packages out there)
I see, but there is no way to write a general training loop from which we could call Optimisers.update with an algorithm like LBFGS.
Optim.jl is completely distinct from Optimisers.jl (maintained by different groups of people). Optim.jl doesn’t provide a way to apply updates like Optimisers sequentially. But in my experience, you can always use callbacks to do most things you need from a custom training loop.