Hello, i am new to julia and using it for numerical simulations. i use a very RAM heavy code that generates multiple dictionaries with a lot of entries. I use Threads.@threads in this code using 60 workers (i have 64 logical proccessors).
After running for a couple of minutes, my CPU usage decreases and becomes periodical:
After some debugging, i realized that this happens after using 80GB of my 512GB RAM, and then julia gets into a loop of short GC sessions. this was confirmed by running the code with GC.gc_enable(false) where GC is disabled and the loops were prevented but a lot of RAM wan needlessly used.
to avoid this, i wanted to use environment variables to change the GC parameters to increase the threshold for starting the GC and making each GC cycle longer so that it will collect all the garbage. i want to avoid the constant start/stop of the multithreading for GC.
i used the following environment variables:
"JULIA_GC_ALLOC_POOL": "400G",
"JULIA_GC_ALLOC_RATIO": "85.0",
"JULIA_GC_PAUSE_ALLOC_PERCENT": "85.0",
"JULIA_GC_PAUSE_ALLOC_BYTES": "400G",
"JULIA_GC_MAX_PAUSE_PERCENT": "85.0",
"JULIA_GC_MAX_PAUSE": "1000000"
in the launch.json file.
to verify these variables, my code runs the line:
println(InteractiveUtils.versioninfo(verbose=true))
which, along other things, prints out:
Environment:
JULIA_GC_ALLOC_POOL = 409600000000
JULIA_GC_ALLOC_RATIO = 85.0
JULIA_NUM_GC_THREADS = 60
JULIA_GC_PAUSE_ALLOC_PERCENT = 85.0
JULIA_GC_PAUSE_ALLOC_BYTES = 400G
JULIA_GC_MAX_PAUSE_PERCENT = 85.0
JULIA_GC_MAX_PAUSE = 1000000
so these variables are passed to the environment.
unfortunately, the code behaves the same after using this environment variables and i still have a big problem with GC. i still get to around 80GB RAM usage with 100% CPU and then the cycles start.
I would appreciate if anyone could suggest how to overcome this problem or perhaps an additional way to debug the problem with using the environment variables.
i would be happy to provide additional data, but unfortunately i’m having a hard time constructing a MWE (cant write a minimal code that is hard enough to keep all cores busy and assign a lot of memory).