# Julia killed with Out of memory error on Linux -- runs fine on MacOS

**URL:** <https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193>\
**Category:** General Usage\
**Tags:** memory, os\
**Created:** [July 28, 2023, 11:06am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193 "2023-07-28T11:06:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nbrantut](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nbrantut/32/51836_2.png) [@nbrantut](https://discourse.julialang.org/u/nbrantut)\
**Post date:** [July 28, 2023, 11:06am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/1 "2023-07-28T11:06:28Z")

</div>

I have a piece of code that runs smoothly on MacOS but keeps crashing on my Linux (ubuntu 22) workstation. The code is quite involved and it is hard to write a meaningful minimal working example for it – it goes through a set of large HDF5 files containing timeseries and computes convolutions with a separate set of signals (this part is computationally expensive), but does not store results or return large arrays (it simply returns a small set of array indices, and pauses to save some text files at some stages).

On a MacOS system (version 12.6), there is no issue and running it on 2 CPUs (super basic parallelisation: each CPU deals with a different file) I can see that each process never requires more than about 1.5Gb memory. The memory requirements do no grow over time and the code executes nicely until all files have been visited.

The exact same code, with same environment and datafiles, gets systematically killed with oom (checked with sudo dmesg) when I run it on a Linux system (Ubuntu 22.04.2 LTS). I have 32 Gb of RAM in both systems, but I can see the memory used by julia processes grow over time when it runs on linux. I have tried going from multiple CPUs to single CPU (removing all use of Distributed), I have tried calling GC.gc() at different stages within the loops, and starting julia with --heap-size-hint=2G. None of this made any substantial difference, and julia gets killed at some early point in each run.

I have seen this issue [https://github.com/JuliaLang/julia/issues/42566](https://github.com/JuliaLang/julia/issues/42566) and this one [https://github.com/JuliaLang/julia/issues/50658](https://github.com/JuliaLang/julia/issues/50658) which sound related, but the discussions there go well above my head.

Note: I remember running that same piece of code a few years back on an older ubuntu version (LTS 18 or so, cannot remember details) and it was doing just perfect. My understanding currently (for what it is worth) is that the issue is rooted in how the OS interacts with Julia (?).

Now the question is: have other users experienced similar issues? And is there any way to (at least temporarily) work around this?

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [July 28, 2023, 11:57am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/2 "2023-07-28T11:57:51Z")

</div>

The second issue you linked seems the most pertinent.

The situation you are encountering sounds like a “memory leak”

One complication may the interaction with HDF5. Are you properly closing your datasets and files so that HDF5 can release memory?

We really could use some more information about your environment.

What is the output of `versioninfo()` and `Pkg.status()` on both macOS and Linux?

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [July 28, 2023, 4:16pm UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/3 "2023-07-28T16:16:26Z")

</div>

I’ve run into similar situations in multithreaded code. Adding regular forces GC fixed the OOM but potentially slowed down the code significantly. This suggests that during multithreaded loops the GC is not working as I expected. This is consistent with getting OOM on a machine with many processors/threads But not on a 2-core system.

Maybe unrelated but I thought worth mentioning?

---

<div class="post-metadata">

**Author:** ![nbrantut](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nbrantut/32/51836_2.png) [@nbrantut](https://discourse.julialang.org/u/nbrantut)\
**Post date:** [July 28, 2023, 4:37pm UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/4 "2023-07-28T16:37:33Z")

</div>

Thanks for your quick answers. Of course I should have checked Julia’s versions. On the macOS, I was using 1.7.2:

```julia
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.2.0)
  CPU: Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, cyclone)

```

and on Ubuntu I was using 1.9.2:

```julia
Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 16 virtual cores

```

The package status output is not necessarily very informative since I am using custom, nonreferenced packages on a private github repository.

Now the question prompted me to try using the same version of Julia (I should have tried this first!), and the good news is that the code runs just fine on 1.7.2 on both systems! I will try to investigate exactly at which version things changed (and if I can reproduce the problem on MacOS too).

Note that calling GC.gc() did not change much the behaviour on Julia 1.9.2 (I did not test exactly if anything changed at all, but I got the same OOM error). Changing to single core operation also produced the same problem…

In any case, thanks for your help!

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [July 28, 2023, 4:48pm UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/5 "2023-07-28T16:48:51Z")

</div>

Can you try to add the following code to those of your functions that allocate a lot:

```julia
if Sys.free_memory()/2^30 < 6.0
    GC.gc()
end

```

This solved a similar issue for me. See also: [OOM despite `--heap-size-hint` · Issue #50658 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/50658)

If this does not solve the issue for you it might be [glibc is optimized for memory allocations microbenchmarks · Issue #42566 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/42566#issuecomment-941335436)

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [July 28, 2023, 6:08pm UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/6 "2023-07-28T18:08:22Z")

</div>

> [@nbrantut](#):
>
> the good news is that the code runs just fine on 1.7.2 on both systems!

Yes, sort of good, but then a regression on the supported 1.9.x. You could try on master or the just released 1.10-beta1. I believe there’s some good work done on the GC, in on master, maybe both. Julia 1.7.2 will of course still work even if no longer officially supported. If you insist on a supported version then you could also try Julia 1.6 LTS. It’s technically still claimed supported, though I believe 1.10 will be the next LTS, possibly soon, and 1.6 then dropped closely after.

I would at least try master (if you want to help Julia development to confirm works there) since I also see:

> <https://github.com/JuliaLang/julia/issues/50704>
>
> We have a set of small benchmarks to quickly test our code in \[RxInfer\](https://…github.com/biaslab/RxInfer.jl). The aim of the package is to run efficient Bayesian inference, potentially on low-power low-memory devices like RaspberryPI. We just noticed, that on Julia 1.10 we have quite a noticeable GC regression. Consider this \[notebook\](https://github.com/biaslab/RxInfer.jl/blob/main/examples/Tiny%20Benchmark.ipynb). Not an MWE but still, this notebook computes Bayesian posteriors in a simple linear Gaussian state-space probabilistic model. There are two settings:
> 
> \- Filtering, for each time step $t$ use observations up to the time step $t$.
> \- Smoothing, for each time step $t$ use observations up to the time step $T \> t$
> 
> Here are the results on the current Julia release
> \`\`\`julia
> julia\> versioninfo()
> Julia Version 1.9.2
> Commit e4ee485e909 (2023-07-05 09:39 UTC)
> 
> julia\> @benchmark run\_filtering($datastream, $n, $v)
> BenchmarkTools.Trial: 1504 samples with 1 evaluation.
> Range (min … max): 2.633 ms … 13.932 ms ┊ GC (min … max): 0.00% … 69.28%
> Time (median): 3.073 ms ┊ GC (median): 0.00%
> Time (mean ± σ): 3.319 ms ± 1.058 ms ┊ GC (mean ± σ): 7.08% ± 13.05%
> 
> ▅▇▇██▇▅▃▂ ▁
> ██████████▇▇▅▇█▇▇▅▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▇██▇▆█▇▆▄▄▅ █
> 2.63 ms Histogram: log(frequency) by time 7.92 ms \<
> 
> Memory estimate: 2.35 MiB, allocs estimate: 63823.
> 
> julia\> @benchmark run\_smoothing($data, $n, $v)
> BenchmarkTools.Trial: 288 samples with 1 evaluation.
> Range (min … max): 13.868 ms … 29.987 ms ┊ GC (min … max): 0.00% … 35.63%
> Time (median): 15.545 ms ┊ GC (median): 0.00%
> Time (mean ± σ): 17.411 ms ± 3.975 ms ┊ GC (mean ± σ): 10.81% ± 14.33%
> 
> ▄▃█▁▄▅▁
> ▇███████▇▆▅▅▃▃▄▂▄▃▂▁▃▃▁▁▁▁▁▁▁▁▁▂▃▅▃▃▅▅▄▃▂▃▃▃▃▃▂▂▁▄▂▁▃▁▁▂▂▂▂ ▃
> 13.9 ms Histogram: frequency by time 28.4 ms \<
> 
> Memory estimate: 10.05 MiB, allocs estimate: 220417.
> \`\`\`
> 
> Here are the results on the 1.10-beta1
> 
> \`\`\`julia
> julia\> versioninfo()
> Julia Version 1.10.0-beta1
> Commit 6616549950e (2023-07-25 17:43 UTC)
> 
> julia\> @benchmark run\_filtering($datastream, $n, $v)
> BenchmarkTools.Trial: 1308 samples with 1 evaluation.
> Range (min … max): 3.260 ms … 78.207 ms ┊ GC (min … max): 0.00% … 94.71%
> Time (median): 3.479 ms ┊ GC (median): 0.00%
> Time (mean ± σ): 3.818 ms ± 3.293 ms ┊ GC (mean ± σ): 6.64% ± 7.41%
> 
> ▄▆██▅▁
> ▂▃▄▇██████▇▅▅▃▃▃▃▃▂▂▃▃▃▃▃▃▂▃▃▂▂▂▁▂▂▂▂▁▂▁▂▂▁▂▂▁▁▁▂▂▂▂▂▂▂▂▂▂ ▃
> 3.26 ms Histogram: frequency by time 4.94 ms \<
> 
> Memory estimate: 2.51 MiB, allocs estimate: 69824.
> 
> julia\> @benchmark run\_smoothing($data, $n, $v)
> BenchmarkTools.Trial: 291 samples with 1 evaluation.
> Range (min … max): 15.160 ms … 88.841 ms ┊ GC (min … max): 0.00% … 79.71%
> Time (median): 15.757 ms ┊ GC (median): 0.00%
> Time (mean ± σ): 17.336 ms ± 7.862 ms ┊ GC (mean ± σ): 7.05% ± 11.57%
> 
> █▅▁
> █████▇▄▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▅▄ ▅
> 15.2 ms Histogram: log(frequency) by time 57.9 ms \<
> 
> Memory estimate: 10.12 MiB, allocs estimate: 222915.
> \`\`\`
> 
> As you can see in the case of \`run\_filtering\`, the maximum time jumped from \`13ms\` to \`78ms\`. The GC max also indicates a jump from \`69%\` to \`94%\`. In the case of \`run\_smoothing\` the situation is similar, the maximum time jumped from \`29ms\` to \`88ms\`. The GC max jumped from \`35%\` to \`79%\`. 
> 
> The inference precedure allocates a lot of intermediate "messages" in a form of distributions from \`Distributions.jl\` package, but \*\*does not\*\* use any sampling. Instead, it computes analytical solutions for posteriors. This analytical solutions also rely on dynamic multiple dispatch in many places. Eliminating dynamic multiple dispatch is not really an option, it just how it works and it was quite efficient anyway until now. 
> 
> The major differences between two functions is that \`run\_filtering\` allocates a lot of information (messages) and do not use it afterwards that is probably can be free-ed right away, and the \`run\_smoothing\` retains/stores this information till the end of the procedure. You can also see that the resulting minimum execution time is also worse in both cases.
> 
> I think this is quite a severe regression, especially for the \`filtering\` case, which is supposed to run the real-time Bayesian inference with as little GC pauses as possible. We can of course refine our code base, but in the mean-time can it be improved in general? What can cause this? How should we proceed and debug this? How can we help figuring out further?
> 
> \`\`\`julia
> julia\> versioninfo()
> Julia Version 1.9.2
> Commit e4ee485e909 (2023-07-05 09:39 UTC)
> Platform Info:
> OS: macOS (x86\_64-apple-darwin22.4.0)
> CPU: 12 × Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
> WORD\_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
> Threads: 1 on 12 virtual cores
> 
> julia\>
> \`\`\`

---

<div class="post-metadata">

**Author:** ![BatyLeo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/batyleo/32/35017_2.png) [@BatyLeo](https://discourse.julialang.org/u/BatyLeo)\
**Post date:** [September 11, 2023, 9:36am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/7 "2023-09-11T09:36:51Z")

</div>

I’m having the same issue since last week, my code runs fine on macos (it uses less than 5G ram) while the ram explodes on ubuntu.  
One of my colleagues seems to also have this issue on a completely different code.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [September 11, 2023, 10:14am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/8 "2023-09-11T10:14:57Z")

</div>

> [@BatyLeo](#):
>
> ram explodes on ubuntu.

Which Julia version? Did you enable zram on Ubuntu?

```julia
sudo apt install zram-config

```

---

<div class="post-metadata">

**Author:** ![BatyLeo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/batyleo/32/35017_2.png) [@BatyLeo](https://discourse.julialang.org/u/BatyLeo)\
**Post date:** [September 12, 2023, 11:14am UTC](https://discourse.julialang.org/t/julia-killed-with-out-of-memory-error-on-linux-runs-fine-on-macos/102193/9 "2023-09-12T11:14:08Z")

</div>

I’m on `v1.9.3`. Switching to `v1.10.0-beta2` seems to fix the issue.