# Memory leak from threading but not pmap

**URL:** <https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839>\
**Category:** General Usage\
**Created:** [May 4, 2019, 2:56am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839 "2019-05-04T02:56:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 4, 2019, 2:56am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/1 "2019-05-04T02:56:38Z")

</div>

I have a function that takes a set of parameters, runs a simulation, and returns 5 values in a named tuple. It does no IO and only calls `rand()` with a seeded RNG that is explicitly passed to hopefully avoid any unsafe threading.

If I run Julia (nightly on linux skylake Xeon) with 4 threads, it will work well for a small number of iterations, but then the memory usage will gradually creep up. Each iteration takes 1-2 seconds, so with the normal GC noise, I expect the memory usage to go up and down by 200MB every few seconds.

For example, in the first few minutes, Julia will use 400-600MB, but by 10 minutes, it’s over 1GB, and by an hour, it’s at 30GB (but still with the 200MB variations).

When I run the same code with `pmap` and 4 workers, each worker stays between 250-310MB with no leakage even after hours.

Any ideas on how to debug what’s going on?

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [May 4, 2019, 3:44am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/2 "2019-05-04T03:44:26Z")

</div>

> [@robsmith11](#):
>
> calls `rand()` with a seeded RNG that is explicitly passed to hopefully avoid any unsafe threading.

I assume you are using a separate RNG object per thread?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [May 4, 2019, 4:27am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/3 "2019-05-04T04:27:54Z")

</div>

Can you post some code/MWE. I have access to a server with a decent amount of RAM that I can test and try to reproduce.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 4, 2019, 11:57am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/4 "2019-05-04T11:57:01Z")

</div>

Yes, I just pass an Int seed as one.of.the parameters and then the RNG is initialized by the function being run in parallel.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 4, 2019, 12:00pm UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/5 "2019-05-04T12:00:20Z")

</div>

> [@affans](#):
>
> Can you post some code/MWE

Thanks, I’ll give it a try but probably not. Unfortunately the simulation that the function runs has thousands of lines of proprietary code.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 4, 2019, 2:50pm UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/6 "2019-05-04T14:50:04Z")

</div>

Ok, I managed to find a very minimal example that eats up 5-10GB of memory within a few seconds:

```julia
Threads.@threads for i in 1:100000
    sum(collect(1:10^6))
end

```

I’m not passing any options to Julia, just setting `JULIA_NUM_THREADS=4`.

Filed an issue here:  
[https://github.com/JuliaLang/julia/issues/31923](https://github.com/JuliaLang/julia/issues/31923)

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [May 5, 2019, 3:45am UTC](https://discourse.julialang.org/t/memory-leak-from-threading-but-not-pmap/23839/7 "2019-05-05T03:45:21Z")

</div>

Meaning you’re explicitly creating a MersenneTwister object (or pcg, whatever your preferred and is), and all your calls to`rand` look like `rand(mt)`?

Edit: although that’s obviously not the problem in your `sum(collect(1:N))` example.