Hi all, Recently, I’m turning to use Julia and Flux.jl to treat some differential equations in computational physics. I find it convenient and efficient indeed. However, I wonder if there is any high-order optimization method can be used to treat these kind of problems with good differentiability…

I have heard of some examples using BFGS on neural networks in Flux. @MikeInnes or @oxinabox is there code on this? I’d like to try it too :slight_smile:.

Flux provides routines to compute the loss function and its gradient, so given those you could throw them at any of the nonlinear optimization packages in Julia that exploit gradients and assume some smoothness (NLopt.jl, Optim.jl, …).

not from me. I have used BFGS in Optim to train NNs without using Flux.

[image] stevengj: Flux provides routines to compute the loss function and its gradient, so given those you could throw them at any of the nonlinear optimization packages in Julia that exploit gradients and assume some smoothness (NLopt.jl, Optim.jl, …). It’s a little hard to do though since F…

Not sure if I understand the problem here, but if the problem is that Optim or any of the other packages expect the parameters of the problem and the gradient to take a specific form/type (i.e. AbstractArray) and this is incompatible with how Flux stores the model, you might want to check out: http…

[image] juthohaegeman: Not sure if I understand the problem here, but if the problem is that Optim or any of the other packages expect the parameters of the problem and the gradient to take a specific form/type (i.e. AbstractArray ) and this is incompatible with how Flux stores the model, you mi…

@vavrines the mixed neural DEs example gives a hint for how to do it: https://github.com/JuliaDiffEq/DiffEqFlux.jl#mixed-neural-des . Essentially the idea is that you need to grab the parameters and manually update the parameters of the neural network. Basically, the secret sauce is that you can us…

Thx Chris :slight_smile: I will try to follow the strategy you mentioned.

Here is a utility package to facilitate using Optims (or most other solvers) (L)BFGS to train flux models [image] [ANN] FluxOptTools Package Announcements FluxOptTools.jl This package contains some utilities to enhance training of Flux.jl models. Train using …

Thank you baggepinnen! I’ll try it.:smiley:

Is there something like ScipyOptimizer in Julia?

General Usage

baggepinnen July 1, 2019, 9:51am 10

Here is a utility package to facilitate using Optims (or most other solvers) (L)BFGS to train flux models

2 Likes

Topic		Replies	Views
Alternative to FluxOptTools? Machine Learning optim , flux , optimization	2	304	March 29, 2023
DiffEqFlux / diffeq_adjoint: no method matching back!(::Float64) Machine Learning diffeq	3	1168	May 28, 2019
Flux differentiation error Machine Learning zygote	19	1796	November 19, 2020
[ANN] FluxOptTools Package Announcements optim , flux , visualization	0	1011	July 1, 2019
How to Approach Parameter Estimation with Non-Linear Optimizers Optimization (Mathematical) question	3	526	December 21, 2022

Is there something like ScipyOptimizer in Julia?

Related topics