# Using LBFGS to train Flux models

**URL:** <https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942>\
**Category:** General Usage\
**Created:** [February 8, 2024, 4:44pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942 "2024-02-08T16:44:55Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gradureau](https://avatars.discourse-cdn.com/v4/letter/g/67e7ee/32.png) [@gradureau](https://discourse.julialang.org/u/gradureau)\
**Post date:** [February 8, 2024, 4:44pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/1 "2024-02-08T16:44:55Z")

</div>

Hey everyone,

I have found out that using LBFGS is quite used in the Physics-Informed Neural Networks and I am wondering it it is interesting to use it for usual MLPs. Is there a way to use it in Flux?

Thank you for your answers!

---

<div class="post-metadata">

**Author:** ![jgreener64](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jgreener64/32/2483_2.png) [@jgreener64](https://discourse.julialang.org/u/jgreener64)\
**Post date:** [February 8, 2024, 10:34pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/2 "2024-02-08T22:34:22Z")

</div>

[FluxOptTools.jl](https://github.com/baggepinnen/FluxOptTools.jl) lets you use Optim with Flux, and Optim has LBFGS.

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [February 9, 2024, 2:24pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/3 "2024-02-09T14:24:20Z")

</div>

Interesting. Do you know if FluxOptTools will work with training on a gpu?

---

<div class="post-metadata">

**Author:** ![jgreener64](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jgreener64/32/2483_2.png) [@jgreener64](https://discourse.julialang.org/u/jgreener64)\
**Post date:** [February 9, 2024, 2:31pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/4 "2024-02-09T14:31:36Z")

</div>

I don’t know, sorry, though it should be quick to test.

---

<div class="post-metadata">

**Author:** ![gradureau](https://avatars.discourse-cdn.com/v4/letter/g/67e7ee/32.png) [@gradureau](https://discourse.julialang.org/u/gradureau)\
**Post date:** [February 9, 2024, 4:32pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/5 "2024-02-09T16:32:33Z")

</div>

Thanks, it works really well. It’s quite impressive to see the efficiency of this optimizer. I wonder why it isn’t teached in ML lectures it’s so great.

Now, one last question : now we use the Optim optimize function that does everything in a black box manner. What is its stopping condition? Is there a way to change it?

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [February 9, 2024, 5:32pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/6 "2024-02-09T17:32:34Z")

</div>

I did try, but ran into trouble, which I didn’t make a serious effort to solve. I will look into it a bit more.

---

<div class="post-metadata">

**Author:** ![jgreener64](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jgreener64/32/2483_2.png) [@jgreener64](https://discourse.julialang.org/u/jgreener64)\
**Post date:** [February 9, 2024, 5:54pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/7 "2024-02-09T17:54:36Z")

</div>

See the [options](https://julianlsolvers.github.io/Optim.jl/stable/#user/config/) available in Optim, e.g. `iterations` and `callback`.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 9, 2024, 6:13pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/8 "2024-02-09T18:13:25Z")

</div>

> [@gradureau](#):
>
> Thanks, it works really well. It’s quite impressive to see the efficiency of this optimizer. I wonder why it isn’t teached in ML lectures it’s so great.

Probably for two reasons. First, in traditional ML you are often doing stochastic optimization, where on each iteration you are sampling a batch of the training data (as opposed to full-batch training), both for efficiency and to avoid over-fitting (see also [this nice lecture on stochastic gradient descent](https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/resources/lecture-25-stochastic-gradient-descent/)), and usually you are converging to relatively low accuracy (again to avoid overfitting — you don’t want a neural net trained to 10 decimal places). There are stochastic L-BFGS variants, but my (vague, possibly wrong) impression is that their advantages over Adam etc. are not as clear as in a non-stochastic high-accuracy setting.

Second, explaining how BFGS works is quite difficult. Even deriving a BFGS step in a basic form requires [a lot of linear algebra and convex-optimization theory](https://github.com/mitmath/18335/blob/spring21/notes/BFGS.pdf), then you have L-BFGS via low-rank approximation, and convergence proofs of BFGS are even harder. Most ML classes don’t assume the necessary background for students to cover any of this in detail.

(But of course, they could mention L-BFGS as a black-box algorithm, handwave a bit about what it does, and show it off in an example.)

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [February 9, 2024, 8:49pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/9 "2024-02-09T20:49:43Z")

</div>

Speculating and spitballing quite a bit, (L)BFGS builds up some cross second derivative information, so it seems plausible that it can move down a local minimum faster than ADAM or similar methods. The limited memory version seems very appropriate for neural nets, which often have very many parameters, compared to many other problems.

It seems conceivable to me that LBFGS could potentially get trapped in a local minimum more easily than more traditional methods for neural nets.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 9, 2024, 9:00pm UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/10 "2024-02-09T21:00:05Z")

</div>

> [@mcreel](#):
>
> Speculating and spitballing quite a bit, (L)BFGS builds up some cross second derivative information, so it seems plausible that it can move down a local minimum faster than ADAM or similar methods.

The caveat here is that the second-derivative information in BFGS becomes more accurate only as you approach a local minimum where the function is approximately quadratic, so sometimes the acceleration only kicks in when you are trying to refine the optimum to higher accuracies (which you often don’t do in typical ML, but do for PINNs). And the benefit might be quickly degraded by noise in stochastic optimization problems.

(The “momentum” terms in Adam etcetera also incorporate some second-derivative information, being closely related to conjugate-gradient methods.)

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [February 10, 2024, 7:25am UTC](https://discourse.julialang.org/t/using-lbfgs-to-train-flux-models/109942/11 "2024-02-10T07:25:14Z")

</div>

The way I see it is that rapidly moving into a local minimum is not really much of a benefit, if you end up trapped there, and that stochastic gradients are an important tool to avoid this happening. So, the performance of LBFGS relative to ADAM, etc, for nets to find a good local min, not just any local min, is probably an empirical question, and it may depend in the starting point.

Regarding second derivatives, the momentum feature of ADAM effectively forms an approximation to own second derivatives, I believe. LBFGS will also incorporate cross partial information, which seems to me to be a notable difference.