Track memory usage

Is there a package/function that I can use to see, track, log, and/or diagnose the memory usage of variables?

I’m serving a website with JSServe and WGLMakie, and after a few hours of usage I get OutOfMemory errors. My initial tests (like printing summarysize) have been helpful, but I’m wondering if there’s something out there that is better-suited for tracking the usage over long periods of time. It would be extra helpful if trends would be picked up as well: like if a variable increases in size by a constant factor as a function of time or a function of “events”.

To be clear, BenchmarkTools is great (use it all the time, thank you @jrevels), but it’s mostly for testing a single function, not the usage of a long process (or perhaps I’m wrong?).

1 Like

You can track allocations using this: https://docs.julialang.org/en/v1/manual/profile/#Memory-allocation-analysis

Maybe that can help, if you leave the process running for a while (or different spans of time), and check where allocations are occurring.

Here I give an example (because the manual is not as direct): Disabling allocations

Depending on your problem, you could check the memory usage and call the GC (I have used this before solving some problems in a parallel code, afterwards it was not useful anymore): https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full

3 Likes

That’s great. It would have been extra useful if that .mem file would have been generated every n minutes, and/or when triggered from inside the code being profiled. This way, I’d be able to diagnose which LOC results in the most obvious increase, not just which LOC needs the most memory.

I guess the easiest way around that (and what I’ll probably be doing next) is to have it running for long enough until it crashes due to OutOfMemory errors, and then the worst LOC should pop up.

Yes. Yet, to be true, after some profiling with that (or not), I generally fall back to the manual memory tracking, with @allocated . I put bunches of code inside something like:

a = @allocated begin
    ....
end; a > 0 && println(a)

You could, when more directed to a point, use that, and print the allocations of some parts of your code to a file whenever that part is executed, and see what is going wrong.

2 Likes

Awesome, thanks a lot for the great pointers.

1 Like

You can also try to use https://github.com/KristofferC/TimerOutputs.jl

It provides more convinient way to trace memory usage and number of calls, you can sprinkle @timeit here and there and print output from time to time.

4 Likes

Oh, I think this might be perfect…

So with the help of Simon Danisch I’m slowly realizing that it’s not allocated memory I need to track, but actually how much of that memory get retained and never released. So I’m back to summarysize

I’m guessing Simon has already told you this, but if not, make sure to update to AbstractPlotting 0.14.2 (which has https://github.com/JuliaPlots/AbstractPlotting.jl/pull/574). That fixed some severe memory issues I was having.

1 Like

Thanks, I am on 0.14.2 :slight_smile:

1 Like

I ended up using all of the solutions mentioned here (so no real one singular way). I’ll argue (for whoever feels up for it) that there is place for a diagnostic tool for detecting memory leaks. Something that could track the change in memory per variable as a function of time/events… But these kind of situations might be too rare to warrant a dedicated tool.

Thanks for all the helpful tips!

2 Likes

So, what was the leak? Something easy to guess with 20/20 hindsight or something unexpected?

I still don’t know. It’s a real mystery…

@JeffreySarnoff recently posted about GFlops.jl (GitHub - triscale-innov/GFlops.jl: Estimate the absolute performance of a piece of Julia code) might not be exactly what you want but I hadn’t seen this package before and it looks pretty solid and useful for diagnosing issues.

Hi, I avoid OutOfMemory errors with the function described here. This works for me on a Linux machine. I frequently call it (in my case in JuMP callbacks), and abort my program if the memory consumption reaches my predefined limit. You can, however, clearly write the return values to a *.log file.
Note that the function returns the current memory usage of your program, not of specific variables. Maybe this helps anyway.