# Why is the memory blowing up in this multi-threaded code?

**URL:** <https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680>\
**Category:** General Usage\
**Created:** [April 3, 2019, 8:25am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680 "2019-04-03T08:25:58Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 8:25am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/1 "2019-04-03T08:25:59Z")

</div>

I have some multi-threaded code in which each thread calls a function `f(df::DataFrame)` which reads a column of that DataFrame and finds the indices where the column is greater than 0:

```julia
function f(df::DataFrame)
    X = df[:time]
    indices = findall(X .> 0)
    indices
end

```

Inside the main thread I read in an R \*.rds file which Julia converts to a DataFrame which I’m passing to `f()` as follows:

```julia
rds = "blabla.rds"
objs = load(rds);

params = collect(0.5:0.005:0.7)

for i in 1:length(objs)
    cols = [string(name) for name in names(objs.data[i]) if occursin("bla",string(name))]
    hypers = [(a,b) for a in cols, b in params] # length ~2000
    Threads.@threads for hi in 1:length(hypers) # MEMORY BLOWS UP HERE
        df = f(df)
    end
end

```

Each `df` that is passed to `f()` is roughly 0.7GB. Analysing the memory usage when the multi-threaded loop is run, the memory usage goes up to ~30GB. There are 25 threads and ~2000 calls to `f()` . Any idea why the memory is exploding?

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 9:55am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/2 "2019-04-03T09:55:16Z")

</div>

Cross-reference: [Julia: Why is the memory blowing up inside this loop? - Stack Overflow](https://stackoverflow.com/questions/55489288/julia-why-is-the-memory-blowing-up-inside-this-loop#55489288)

(Just so you know, it’s considered good etiquette to mention it if you have posted the question other places.)

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 9:59am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/3 "2019-04-03T09:59:40Z")

</div>

Try re-writing like this:

```julia
function f(df::DataFrame)
    X = df[:time]
    return findall(x->x>0, X)
end

function foo(objs)
    params = 0.5:0.005:0.7 # don't collect
    for i in 1:length(objs)
        cols = [string(name) for name in names(objs.data[i]) if occursin("bla",string(name))]
        hypers = [(a,b) for a in cols, b in params] # length ~2000
        Threads.@threads for hi in 1:length(hypers) # MEMORY BLOWS UP HERE
            df = f(df)
        end
    end
end

using BenchmarkTools

rds = "blabla.rds"
objs = load(rds);
@benchmark foo($objs)

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 3, 2019, 10:06am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/4 "2019-04-03T10:06:20Z")

</div>

Instead of one thread allocating, you have 20 threads allocating. FWIW, I have almost never seen a speedup from multithreading if the part that is running parallel is allocating. The GC runs on one thread so if you have 20 threads allocating, it will bottleneck things.

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 1:25pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/5 "2019-04-03T13:25:33Z")

</div>

```julia
BenchmarkTools.Trial: 
  memory estimate: 35.09 GiB
  allocs estimate: 59882
  --------------
  minimum time: 2.778 s (0.00% GC)
  median time: 2.801 s (0.00% GC)
  mean time: 2.801 s (0.00% GC)
  maximum time: 2.825 s (0.00% GC)
  --------------
  samples: 2
  evals/sample: 1

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 1:50pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/6 "2019-04-03T13:50:58Z")

</div>

So is this threaded or not? Any chance you can provide a MWE? Right now we’re mainly guessing.

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 1:55pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/7 "2019-04-03T13:55:11Z")

</div>

```julia
rds = "bla.rds"
objs = load(rds);

function f(df::DataFrame)
    X = df[:time]
    return findall(x->x>0, X)
end

function foo(objs)
    for i in 1:length(objs)
        df = objs.data[i]
        Threads.@threads for hi in 1:2000
            f(df)
        end
    end
end

@benchmark(foo($objs))

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 1:56pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/8 "2019-04-03T13:56:20Z")

</div>

Thanks, but we don’t have any data to run this on. A toy dataset would work.

What sort of memory use are you expecting? How big is the data set?

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 1:57pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/9 "2019-04-03T13:57:33Z")

</div>

objs is around 1GB.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 1:58pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/10 "2019-04-03T13:58:48Z")

</div>

So since you iterate 2000 times, conceivably you would expect anywhere up to 2TB of memory use?

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 2:00pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/11 "2019-04-03T14:00:26Z")

</div>

OK. I guess I’m asking why is the dataframe not being shared across threads? I thought it’s passed by reference?

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 3, 2019, 2:01pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/12 "2019-04-03T14:01:33Z")

</div>

The same idea in Numba does not blow up the memory.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 2:03pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/13 "2019-04-03T14:03:26Z")

</div>

I guess it is, but you have a bunch of index arrays being allocated in your loop.

Some toy data with Numba and Julia code would be a big help.

(Gotta run)

---

<div class="post-metadata">

**Author:** ![Rui](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@Rui](https://discourse.julialang.org/u/Rui)\
**Post date:** [April 3, 2019, 2:39pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/14 "2019-04-03T14:39:02Z")

</div>

Interesting. Do you know if it is possible and planned to make it the case that each thread handles its garbage?

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 4:08pm UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/15 "2019-04-03T16:08:10Z")

</div>

Is there really anything unexpected going on here, at all? To me it seems like a code that allocates one `Vector` of indices per iteration. The memory use is the size of the index vector times the number of iterations. If you want lower memory consumption you need to reuse memory between iterations.

What do the Numba benchmarks look like? Are you sure they are reporting the same thing as here?

Cutting out the dataframe and just working with vectors give me the same memory use.

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 4, 2019, 7:40am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/16 "2019-04-04T07:40:35Z")

</div>

What’s a good way to get equivalent benchmarks in Numba?

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 4, 2019, 7:47am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/17 "2019-04-04T07:47:19Z")

</div>

Also, I ran foo() with just a single array, and the memory is still blowing up:

```julia
function f(time::Array{Float64,1})
    return findall(x->x>0, time)
end

function foo(time::Array{Float64,1})
    Threads.@threads for hi in 1:2000
        f(time)
    end
end

@benchmark(foo($objs.data[1][:time]))

```

The array is 1631339 rows.

I’m running the equivalent code in Numba as follows:

```julia
@jit(nopython=True)
def f(time):
    return np.where(time > 0)[0]

@jit(nopython=True, parallel=True)
def foo(time):
    for h in prange(2000):
        f(time)

foo(df['time'].values)

```

I’m not sure how to get the equivalent of @benchmark in Numba, but looking at htop, the memory doesn’t blow up anywhere near what it does with Julia.

This is not a jupyter notebook thing either; running a pure julia script gives the same result.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 4, 2019, 7:59am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/18 "2019-04-04T07:59:38Z")

</div>

> [@ruthmarx1](#):
>
> and the memory is still blowing up:

What do you mean by this? The number reported is the sum of all allocations done during the execution. So if you disabled the GC completely you would have 32 GB of memory allocated. Fortunately, we do have a GC.

---

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 4, 2019, 8:07am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/19 "2019-04-04T08:07:11Z")

</div>

The RES memory in htop increases and doesn’t go back to what it was before I ran the function. I’m not seeing this in Numba.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 4, 2019, 8:20am UTC](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680/20 "2019-04-04T08:20:23Z")

</div>

Could you post your code as text instead of images so that people that want to try it out don’t have to type the whole thing in manually?

[Next page](https://discourse.julialang.org/t/why-is-the-memory-blowing-up-in-this-multi-threaded-code/22680.md?page=2)