Is there a package/function that I can use to see, track, log, and/or diagnose the memory usage of variables?
I’m serving a website with JSServe and WGLMakie, and after a few hours of usage I get OutOfMemory errors. My initial tests (like printing summarysize) have been helpful, but I’m wondering if there’s something out there that is better-suited for tracking the usage over long periods of time. It would be extra helpful if trends would be picked up as well: like if a variable increases in size by a constant factor as a function of time or a function of “events”.
To be clear, BenchmarkTools is great (use it all the time, thank you @jrevels), but it’s mostly for testing a single function, not the usage of a long process (or perhaps I’m wrong?).
That’s great. It would have been extra useful if that .mem file would have been generated every n minutes, and/or when triggered from inside the code being profiled. This way, I’d be able to diagnose which LOC results in the most obvious increase, not just which LOC needs the most memory.
I guess the easiest way around that (and what I’ll probably be doing next) is to have it running for long enough until it crashes due to OutOfMemory errors, and then the worst LOC should pop up.
Yes. Yet, to be true, after some profiling with that (or not), I generally fall back to the manual memory tracking, with @allocated . I put bunches of code inside something like:
a = @allocated begin
....
end; a > 0 && println(a)
You could, when more directed to a point, use that, and print the allocations of some parts of your code to a file whenever that part is executed, and see what is going wrong.
So with the help of Simon Danisch I’m slowly realizing that it’s not allocated memory I need to track, but actually how much of that memory get retained and never released. So I’m back to summarysize…
I ended up using all of the solutions mentioned here (so no real one singular way). I’ll argue (for whoever feels up for it) that there is place for a diagnostic tool for detecting memory leaks. Something that could track the change in memory per variable as a function of time/events… But these kind of situations might be too rare to warrant a dedicated tool.
Hi, I avoid OutOfMemory errors with the function described here. This works for me on a Linux machine. I frequently call it (in my case in JuMP callbacks), and abort my program if the memory consumption reaches my predefined limit. You can, however, clearly write the return values to a *.log file.
Note that the function returns the current memory usage of your program, not of specific variables. Maybe this helps anyway.
