Optim Consumes an Increasingly Large Amount of Memory (Until it Crashes)


I am using Optim using LBFGS for a statistical application (maximum likelihood), and I have encountered the issue that Optim consumes an increasingly large amount of memory until it runs out and crashes. The data set size is on the order of 1-2GB, and there are ~1000 parameters. I tried running Optim and watching memory use over the iterations. Weirdly, the memory did go up and then down again (presumably with garbage collection) until iteration ~70 or so, after which point it continued to consume memory until it crashed (using up all ~155GB available). It’s worth noting that I provide an analytic gradient, so it’s not ForwardDiff.Duals that are generating the memory use.

Any thoughts about what could be going on or how to avoid/resolve the issue would be greatly appreciated. Thank you!


Well, it would certainly make it a lot easier if you showed us the optimize call :slight_smile:

Edit: to make it even more easy to help you, you could maybe fake some data and show us everything.


Have you tried reducing the value of m where m is the number of prior steps that it records (see docs)? I assume you haven’t set it to 70? That would it explain your problem nicely.

edit: I accidentally included a quote.


@jcook, I left it at the default of m=10. I would imagine that LBFGS doesn’t hold onto the old values more than m.

@pkofod, it’s a relatively complicated model. I will see if I can code up an MWE.


That’s fine, I can abstract away from f and g! for now, I just want to see what you do exactly wrt the optimize call. Do you set any options for example?


@pkofod, I investigated further, and it appears that the issue is not Optim. It appears that there is a substantial memory leak in my function. Even outside of Optim, calling the gradient a couple hundred times in a loop causes a crash. I’m running a few more tests and putting together an MWE, which I will post to another thread, since this does not appear to be an optimization issue.

Thank you both for your help!


Good to see that you’re on track to find the issue. When it doesn’t crash anymore, don’t hesitate asking any other Optim questions you may have.