What's using up so much memory?

General Usage
1

After loading some data into a DataFrame using LibPQ, I can see from htop that the Julia process is using 9.5 GB of resident memory, which doesn’t shrink even after running the garbage collector.

I only have 1.2 GB of variables in my workspace. For what is the extra 8.3 GB being used and is it possible to reclaim it?

  name                    size summary
  –––––––––––––––– ––––––––––– ––––––––––––––––––––––––––––––––––––––––––––––
  Base                         Module
  Core                         Module
  InteractiveUtils 261.186 KiB Module
  Main                         Module
  NA                   8 bytes Float64
  ans                4.750 KiB Markdown.MD
  irank                0 bytes irank (generic function with 1 method)
  p                  1.208 GiB 64839923×3 DataFrame
  run1                 0 bytes run1 (generic function with 1 method)
  select_prices        0 bytes select_prices (generic function with 1 method)

julia> versioninfo()
Julia Version 1.8.0-DEV.709
Commit 1389c2fc4a (2021-10-12 16:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen 7 4700U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)
Environment:
  JULIA_DEPOT_PATH = /me/.julia
1 Like
2

Can you try running the garbage collector more than once? Sometimes that changes things (don’t ask me why).

Maybe at some earlier point your program was using all that memory and what is happening is just that the OS has not reclaimed it?

1 Like
3

Running gc() multiple times had no effect.

I opened another process and gradually filled the system’s memory. The memory usage of the original Julia process eventually went down as the memory filled up, but not until swapping started.

Maybe there are some linux kernel parameters that can be tweaked to get it to reclaim memory more aggressively. It makes it difficult to avoid swapping or even running out of memory when I can’t get a reasonable estimate of how much memory Julia is actually using.

4

Could it be related to this? https://github.com/JuliaData/CSV.jl/issues/850

1 Like
5

Did you ever figure this one out? Reason I’m asking is I also have a program using LibPQ (periodically pulls data from one location to put it into a DB) which swells in memory as time goes. Committed memory in Windows grows from 3GB to 50GB after a few hours of use.
I’ve tried to insert GC calls in my loops, added proper destruction of objects by assigning Nothing, to no avail.

6

I saw the 1.8.3 and got immediately excited, thinking it would solve my issue. So on the bright side, the allocations reported by the @time call within the loop are much smaller, but on the not so bright side the committed memory grows roughly by 5MB/s which is a much bigger pace than previously.
So something in the accounting of @time may have changed, and something else sure looks like it’s leaking more than before.

7

It hasn’t been an issue for any of my more recent projects, so not sure if the situation changed at all.

Have you already seen this issue? It might be relevant.
https://github.com/JuliaLang/julia/issues/42566

8

Thank you for linking this. I’m not sure it’s the culprit, especially as I’m running my sample code in Windows 10. I would try it under Linux but one of the pieces of my short program is Windows only.

I am creating lots of (temporary) dataframes though, which contain String, Float, DateTime and ZonedDateTime columns. I’ve thoroughly inserted assignments to Nothing whenever possible to help the GC, as well as forced GC calls regularly.

After running from say 7am to 5pm, said program uses 1.8 GB in so-called active memory (Win 10) and 16.8 GB in committed. Not sure how I can expect memory within the Julia instance though, to see which objects remain there.

9

Interestingly, after letting the above Julia REPL rest overnight, active memory under Windows dropped all the way to 2MB while committed memory stayed at 16.8GB.

And a simple GC.gc() call takes it back to 255MB.

10

Windows swapped out pages because the program wasn’t doing anything.

2 Likes
11

Seems like julia was releasing the memory but Windows was not reclaiming it.

12

The committed memory remains at its high watermark though.

Running the same routing today I reached 70GB of committed and other applications started crashing as I was running out of total memory.

13

It seems that with the 1.9 betas the problem disappears. Or at least is not as bad. The same program as above seems to stay under 5GB instead of ballooning to 70GB of committed memory.

1 Like