Optim Consumes an Increasingly Large Amount of Memory (Until it Crashes)

Specific Domains Optimization (Mathematical)
1

I am using Optim using LBFGS for a statistical application (maximum likelihood), and I have encountered the issue that Optim consumes an increasingly large amount of memory until it runs out and crashes. The data set size is on the order of 1-2GB, and there are ~1000 parameters. I tried running Optim and watching memory use over the iterations. Weirdly, the memory did go up and then down again (presumably with garbage collection) until iteration ~70 or so, after which point it continued to consume memory until it crashed (using up all ~155GB available). It’s worth noting that I provide an analytic gradient, so it’s not ForwardDiff.Duals that are generating the memory use.

Any thoughts about what could be going on or how to avoid/resolve the issue would be greatly appreciated. Thank you!

2

Well, it would certainly make it a lot easier if you showed us the optimize call :slight_smile:

Edit: to make it even more easy to help you, you could maybe fake some data and show us everything.

3

Have you tried reducing the value of m where m is the number of prior steps that it records (see docs)? I assume you haven’t set it to 70? That would it explain your problem nicely.

edit: I accidentally included a quote.

4

@jcook, I left it at the default of m=10. I would imagine that LBFGS doesn’t hold onto the old values more than m.

@pkofod, it’s a relatively complicated model. I will see if I can code up an MWE.

5

That’s fine, I can abstract away from f and g! for now, I just want to see what you do exactly wrt the optimize call. Do you set any options for example?

6

@pkofod, I investigated further, and it appears that the issue is not Optim. It appears that there is a substantial memory leak in my function. Even outside of Optim, calling the gradient a couple hundred times in a loop causes a crash. I’m running a few more tests and putting together an MWE, which I will post to another thread, since this does not appear to be an optimization issue.

Thank you both for your help!

7

Good to see that you’re on track to find the issue. When it doesn’t crash anymore, don’t hesitate asking any other Optim questions you may have.