# Memory blow-up when passing DataFrame to function inside @threads loop

**URL:** <https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643>\
**Category:** Julia at Scale\
**Created:** [April 2, 2019, 3:16pm UTC](https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643 "2019-04-02T15:16:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ruthmarx1](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Post date:** [April 2, 2019, 3:16pm UTC](https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643/1 "2019-04-02T15:16:55Z")

</div>

When I run the code below, the memory usage blows up:

```julia
function func(df::DataFrame)
    X = df[:time]
    indices = findall(X .> 0)
end

# read in R data
rds = "blablab.rds"
objs = load(rds);

params = collect(0.5:0.005:0.7);

for i in 1:length(objs)
    cols = [string(name) for name in names(objs.data[i]) if occursin("blabla",string(name))]
    hypers = [(a,b) for a in cols, b in params]

    results = [DataFrame() for _ in 1:length(hypers)]

    # HERE IS WHERE THE MEMORY BLOWS UP
    Threads.@threads for hi in 1:length(hypers)
        name, val = hypers[hi]
        results[hi] = func(objs.data[i])
    end
end

```

`df` is 0.7GB. When I run this piece of code my memory usage goes up to ~30GB!!! It seems like just accessing a column of `df` inside `func()` is copying the whole thing?

---

<div class="post-metadata">

**Author:** ![Dave\_Marchant](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave_marchant/32/7882_2.png) [@Dave\_Marchant](https://discourse.julialang.org/u/Dave_Marchant)\
**Post date:** [April 2, 2019, 5:06pm UTC](https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643/2 "2019-04-02T17:06:15Z")

</div>

It looks like it could be similar to the issue that I ran into here:

> <https://github.com/JuliaData/DataFrames.jl/issues/1679>
>
> I'm having a problem working with Distributed.RemoteChannel any time DataFrames …is loaded. I've come up with this minimal example to reproduce the problem:
> 
> \`\`\`
> using DataFrames
> using Distributed
> addprocs(1)
> out = RemoteChannel(()-\>Channel(1), 2)
> println(out)
> \`\`\`
> 
> on mac, this results in:
> \`\`\`
> ERROR: LoadError: StackOverflowError:
> deserialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Type{RemoteChannel{Channel{Any}}}) at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:310 (repeats 100 times)
> Stacktrace:
> \[1\] #remotecall\_fetch#149(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:379
> \[2\] remotecall\_fetch(::Function, ::Distributed.Worker, ::Function, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:371
> \[3\] #remotecall\_fetch#152(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Function, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:406
> \[4\] remotecall\_fetch at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:406 \[inlined\]
> \[5\] RemoteChannel(::Function, ::Int64) at /Users/osx/buildbot/slave/package\_osx64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:108
> \[6\] top-level scope at none:0
> \[7\] include at ./boot.jl:317 \[inlined\]
> \[8\] include\_relative(::Module, ::String) at ./loading.jl:1044
> \[9\] include(::Module, ::String) at ./sysimg.jl:29
> \[10\] exec\_options(::Base.JLOptions) at ./client.jl:266
> \[11\] \_start() at ./client.jl:425
> \`\`\`
> 
> Removing the \`using DataFrames\` runs no problem. The way it is crashes seems to vary depending on the platform I'm running it on. On linux (Ubuntu 16.04.4) it does not crash, but hangs on the RemoteChannel line while slowly ramping memory use to 100%.

If it is the same issue, I believe it was [fixed](https://github.com/JuliaLang/julia/issues/30679) on the julia master branch a few months ago, but hasn’t made it into a release yet. Again, if it is the same problem, things should work with julia 1.0.3 and earlier. I’ve been sticking with that release until we get a new bug fix release.