What's using up so much memory?

After loading some data into a DataFrame using LibPQ, I can see from htop that the Julia process is using 9.5 GB of resident memory, which doesn’t shrink even after running the garbage collector.

I only have 1.2 GB of variables in my workspace. For what is the extra 8.3 GB being used and is it possible to reclaim it?

  name                    size summary
  –––––––––––––––– ––––––––––– ––––––––––––––––––––––––––––––––––––––––––––––
  Base                         Module
  Core                         Module
  InteractiveUtils 261.186 KiB Module
  Main                         Module
  NA                   8 bytes Float64
  ans                4.750 KiB Markdown.MD
  irank                0 bytes irank (generic function with 1 method)
  p                  1.208 GiB 64839923×3 DataFrame
  run1                 0 bytes run1 (generic function with 1 method)
  select_prices        0 bytes select_prices (generic function with 1 method)

julia> versioninfo()
Julia Version 1.8.0-DEV.709
Commit 1389c2fc4a (2021-10-12 16:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen 7 4700U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)
Environment:
  JULIA_DEPOT_PATH = /me/.julia
1 Like

Can you try running the garbage collector more than once? Sometimes that changes things (don’t ask me why).

Maybe at some earlier point your program was using all that memory and what is happening is just that the OS has not reclaimed it?

1 Like

Running gc() multiple times had no effect.

I opened another process and gradually filled the system’s memory. The memory usage of the original Julia process eventually went down as the memory filled up, but not until swapping started.

Maybe there are some linux kernel parameters that can be tweaked to get it to reclaim memory more aggressively. It makes it difficult to avoid swapping or even running out of memory when I can’t get a reasonable estimate of how much memory Julia is actually using.

Could it be related to this? Memory used during parsing never reclaimed · Issue #850 · JuliaData/CSV.jl · GitHub

1 Like