I have a script for calculation of millions of strings distances with StringDistances.jl. I’m using 64 CPUs virtual machine with 240GB RAM and found that my script was killed by out of memory. I cleared the script to minimize object allocations but Julia still takes up to 240GB. Memory consumption in time is far from linear but rather a saw. Looks like garbage collector is switched on after few minutes but next is completely switched off. Also, there is dependency on how many threads I’m running. When the threads number is much less than number of available CPUs it works. E.g. 30 from 64. But when I’m running the script with 60 threads on 64 CPUs VM, I’m getting out of memory after some time. The only way I found how to finish the calculation properly is to add explicit memory control:
@threads for item in list # do something useful with `item` # In my case this part is calculated few minutes # ... if (Sys.free_memory() / Sys.total_memory() < 0.1) GC.gc() sleep(10) end end
And running with
JULIA_NUM_THREADS=60 julia --project=@. src/...
After ~10 hours of calculation with 60 threads I got results. And I can say that my real script’s memory consumption is less that 5-10 GB but not 240 GB of available RAM.
I found similar issue https://github.com/JuliaLang/julia/issues/6103 but looks it is still actual.
Julia 1.2, CentOS 7
So, the questions are how to avoid that explicit code in my script and does Julia do automatic cleaning of memory instead of collecting garbage and be killed by operational system by out of memory?