# Optim Consumes an Increasingly Large Amount of Memory (Until it Crashes)

**URL:** <https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689>\
**Category:** Optimization (Mathematical)\
**Created:** [July 27, 2018, 7:35pm UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689 "2018-07-27T19:35:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juser](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@Juser](https://discourse.julialang.org/u/Juser)\
**Post date:** [July 27, 2018, 7:35pm UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/1 "2018-07-27T19:35:07Z")

</div>

I am using Optim using LBFGS for a statistical application (maximum likelihood), and I have encountered the issue that Optim consumes an increasingly large amount of memory until it runs out and crashes. The data set size is on the order of 1-2GB, and there are ~1000 parameters. I tried running Optim and watching memory use over the iterations. Weirdly, the memory did go up and then down again (presumably with garbage collection) until iteration ~70 or so, after which point it continued to consume memory until it crashed (using up all ~155GB available). It’s worth noting that I provide an analytic gradient, so it’s not ForwardDiff.Duals that are generating the memory use.

Any thoughts about what could be going on or how to avoid/resolve the issue would be greatly appreciated. Thank you!

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [July 28, 2018, 9:34am UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/2 "2018-07-28T09:34:53Z")

</div>

> [@Juser](#):
>
> Any thoughts about what could be going on or how to avoid/resolve the issue would be greatly appreciated. Thank you!

Well, it would certainly make it a lot easier if you showed us the `optimize` call 🙂

Edit: to make it even more easy to help you, you could maybe fake some data and show us everything.

---

<div class="post-metadata">

**Author:** ![jcook](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jcook/32/18211_2.png) [@jcook](https://discourse.julialang.org/u/jcook)\
**Post date:** [July 28, 2018, 5:31pm UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/3 "2018-07-28T17:31:19Z")

</div>

Have you tried reducing the value of `m` where `m` is the number of prior steps that it records (see [docs](https://github.com/JuliaNLSolvers/Optim.jl/blob/master/docs/src/algo/lbfgs.md))? I assume you haven’t set it to 70? That would it explain your problem nicely.

edit: I accidentally included a quote.

---

<div class="post-metadata">

**Author:** ![Juser](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@Juser](https://discourse.julialang.org/u/Juser)\
**Post date:** [July 28, 2018, 6:25pm UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/4 "2018-07-28T18:25:55Z")

</div>

@jcook, I left it at the default of `m=10`. I would imagine that LBFGS doesn’t hold onto the old values more than `m`.

@pkofod, it’s a relatively complicated model. I will see if I can code up an MWE.

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [July 28, 2018, 8:06pm UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/5 "2018-07-28T20:06:58Z")

</div>

That’s fine, I can abstract away from `f` and `g!` for now, I just want to see what you do exactly wrt the `optimize` call. Do you set any options for example?

---

<div class="post-metadata">

**Author:** ![Juser](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@Juser](https://discourse.julialang.org/u/Juser)\
**Post date:** [July 29, 2018, 4:50am UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/6 "2018-07-29T04:50:32Z")

</div>

@pkofod, I investigated further, and it appears that the issue is not Optim. It appears that there is a substantial memory leak in my function. Even outside of Optim, calling the gradient a couple hundred times in a loop causes a crash. I’m running a few more tests and putting together an MWE, which I will post to another thread, since this does not appear to be an optimization issue.

Thank you both for your help!

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [July 29, 2018, 7:19am UTC](https://discourse.julialang.org/t/optim-consumes-an-increasingly-large-amount-of-memory-until-it-crashes/12689/7 "2018-07-29T07:19:56Z")

</div>

Good to see that you’re on track to find the issue. When it doesn’t crash anymore, don’t hesitate asking any other Optim questions you may have.