# DataFrame type instability - Unneccesary memory allocation

**URL:** <https://discourse.julialang.org/t/dataframe-type-instability-unneccesary-memory-allocation/22465>\
**Category:** Data\
**Created:** [March 28, 2019, 3:52pm UTC](https://discourse.julialang.org/t/dataframe-type-instability-unneccesary-memory-allocation/22465 "2019-03-28T15:52:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eric\_Chen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_chen/32/5992_2.png) [@Eric\_Chen](https://discourse.julialang.org/u/Eric_Chen)\
**Post date:** [March 28, 2019, 3:52pm UTC](https://discourse.julialang.org/t/dataframe-type-instability-unneccesary-memory-allocation/22465/1 "2019-03-28T15:52:25Z")

</div>

I am running Julia 1.1, here is my Code

```julia
using BenchmarkTools
using DataFrames

df = DataFrame(rand(10_000_000, 10));
Threads.nthreads()

function singlethread(df)
    N = nrow(df)
    M = ncol(df)

    cumsum = 0.0
    for i in 1:N
        for j in 1:M
            cumsum += df[j][i]
        end
    end
    return cumsum
end

function multithread(df)
    N = nrow(df)
    M = ncol(df)

    cumsum = Threads.Atomic{Float64}(0.0)
    Threads.@threads for j in 1:M
        for i in 1:N
            Threads.atomic_add!(cumsum, df[j][i])
        end
    end
    return cumsum[]
end

@btime singlethread(df)
@btime multithread(df)

```

My Result:

Threads.nthreads()  
4

@btime singlethread(df)  
3.898 s (299994891 allocations: 4.47 GiB)

@btime multithread(df)  
5.845 s (143212953 allocations: 2.10 GiB)

There are unnecessary memory allocation, because of I did “df[j][i]” right?  
If my function has to read the df using a format of df[j][i] or df[i, j],  
e.g. my function has to read the data one by one, row by row, column by column, to perform some operations, then how can i avoid type instability?

Below, I convert everything in a 2D array, then I have almost 0 memory allocation which is good!

```julia
using BenchmarkTools
arr = rand(10_000_000, 10);
Threads.nthreads()

function singlethread(arr)
    N, M = size(arr)

    cumsum = 0.0
    for i in 1:N
        for j in 1:M
            cumsum += arr[i, j]
        end
    end
    return cumsum
end

function multithread(arr)
    N, M = size(arr)

    cumsum = Threads.Atomic{Float64}(0.0)
    Threads.@threads for j in 1:M
        for i in 1:N
            Threads.atomic_add!(cumsum, arr[i, j])
        end
    end
    return cumsum[]
end

@btime singlethread(arr)
@btime multithread(arr)

```

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [March 28, 2019, 5:27pm UTC](https://discourse.julialang.org/t/dataframe-type-instability-unneccesary-memory-allocation/22465/2 "2019-03-28T17:27:34Z")

</div>

The solution is to pass column vectors to the function as a tuple or named tuple (e.g. using `singlethread(tuple(eachcol(df, false)...))`) so that it will be specialized on the particular types. If all columns have the same types, you can also pass a vector of columns to avoid specializing on the number of columns (but be careful that it’s concretely typed).