# Garbage collector behaviour when memory is almost full
**URL:** https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169
**Category:** Performance
**Created:** [August 29, 2019, 3:39pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169 "2019-08-29T15:39:17Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [August 29, 2019, 3:39pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/1 "2019-08-29T15:39:17Z")
I have a script for calculation of millions of strings distances with StringDistances.jl. I’m using 64 CPUs virtual machine with 240GB RAM and found that my script was killed by out of memory. I cleared the script to minimize object allocations but Julia still takes up to 240GB. Memory consumption in time is far from linear but rather a saw. Looks like garbage collector is switched on after few minutes but next is completely switched off. Also, there is dependency on how many threads I’m running. When the threads number is much less than number of available CPUs it works. E.g. 30 from 64. But when I’m running the script with 60 threads on 64 CPUs VM, I’m getting out of memory after some time. The only way I found how to finish the calculation properly is to add explicit memory control:
```julia
@threads for item in list
# do something useful with `item`
# In my case this part is calculated few minutes
# ...
if (Sys.free_memory() / Sys.total_memory() < 0.1)
GC.gc()
sleep(10)
end
end
```
And running with `JULIA_NUM_THREADS=60 julia --project=@. src/...`
After ~10 hours of calculation with 60 threads I got results. And I can say that my real script’s memory consumption is less that 5-10 GB but not 240 GB of available RAM.
I found similar issue [https://github.com/JuliaLang/julia/issues/6103](https://github.com/JuliaLang/julia/issues/6103) but looks it is still actual.
Julia 1.2, CentOS 7
So, the questions are how to avoid that explicit code in my script and does Julia do automatic cleaning of memory instead of collecting garbage and be killed by operational system by out of memory?
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [August 29, 2019, 5:23pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/2 "2019-08-29T17:23:10Z")
while julia for sure should handle this, I just want to ask some simple question since no MWE is provided, can you pre-allocate? if all 60 threads are all comparing the biggest stings, what’s the estimated memory consumption?
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [August 29, 2019, 6:30pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/3 "2019-08-29T18:30:34Z")
I noticed the following issue posted to Julia’s issue tracker about GC under multithreaded situations: [https://github.com/JuliaLang/julia/issues/33097](https://github.com/JuliaLang/julia/issues/33097)
Essentially, because the GC pass requires all threads to hit a safepoint (and then pause), it’s possible that one or more “runaway” threads keep the GC pass from occurring, and so those threads can keep accumulating garbage that isn’t GC’d before hitting the memory limit. The PR linked in that issue ([https://github.com/JuliaLang/julia/pull/33092](https://github.com/JuliaLang/julia/pull/33092)) provides a means to explicitly do a cheap safepoint check in the middle of compute-bound code. Maybe it’s worth giving that PR a try?
---
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [August 29, 2019, 7:19pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/4 "2019-08-29T19:19:41Z")
@jling, yes, I’m pre-allocating and even pre-processing strings before doing distance calculations. Actually the task is elementary. Simply speaking I have two lists of strings with ~10-30 characters. In the mentioned case with 10h x 60 threads it was 20k + 70k strings. And I’m using RatcliffObershelp() metrics for distance calculation. Even sorting of tokens I’m doing once for these stings. 60 threads are going through 20k list.
@jpsamaroo, thanks. It looks like I used old style but almost proper way how to fix it temporarily 🙂 The task where I found the issue is a single run task. So, definitely I can check it with mentioned by you safepoint. At the same time I hope it will be fixed soon.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [August 15, 2020, 5:35pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/5 "2020-08-15T17:35:21Z")
> [@rssdev10](#):
>
> ```julia
> if (Sys.free_memory() / Sys.total_memory() < 0.1)
> GC.gc()
> sleep(10)
> end
>
> ```
I had the same type of problem with a parallel computation of mine. The default garbage collection of Julia was not being safe enough when many threads were being launched.
This workaround solved the problem completely, but why did you add the `sleep(10)` directive?
---
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [August 15, 2020, 8:34pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/6 "2020-08-15T20:34:11Z")
The only reason why I added sleep() is to release a current processor time slice scheduled by an operational system. 10 is less than a typical time slice. In that case, I’m expecting that a thread with a garbage collector will be able to do something useful.
---
### Author:  [@danrib07](https://discourse.julialang.org/u/danrib07)
#### Post date: [June 23, 2021, 8:06pm UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/7 "2021-06-23T20:06:47Z")
Hi,
I think this thread can be helpful on an issue I’m working on with segfaults in julia. However I did not quite understand the purpose of `sleep(10)`. Could you elaborate on it a little more?
Thank you!!
---
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [June 24, 2021, 4:59am UTC](https://discourse.julialang.org/t/garbage-collector-behaviour-when-memory-is-almost-full/28169/8 "2021-06-24T04:59:15Z")
> **[Scheduling (computing)](https://en.wikipedia.org/wiki/Scheduling_(computing))**
>
> In computing, scheduling is the action of assigning resources to perform tasks. The resources may be processors, network links or expansion cards. The tasks may be threads, processes or data flows.
> The scheduling activity is carried out by a process called scheduler. Schedulers are often designed so as to keep all computer resources busy (as in load balancing), allow multiple users to share system resources effectively, or to achieve a target quality-of-service.
> Scheduling is fundamental to com...
When a thread is doing sleep, it means that it is not participating in the tasks scheduling. 10 milliseconds is less than minimal time slice in most operational systems but it is enough to switch a processor core to another thread. The garbage collection issue is happening when a system is high loaded. So, excluding the current computational thread from the cycle of task scheduling increases probability of finishing utility work by auxiliary threads. Including the background garbage collection.