# Would somebody suggest what is the right to work in a parallel loop and write data to the same Dataframe?

**URL:** <https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949>\
**Category:** New to Julia\
**Created:** [January 4, 2020, 12:21am UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949 "2020-01-04T00:21:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [January 4, 2020, 12:21am UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949/1 "2020-01-04T00:21:16Z")

</div>

Would somebody suggest what is the right to work in a parallel loop and write data to the same Dataframe?

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [January 4, 2020, 7:40pm UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949/2 "2020-01-04T19:40:38Z")

</div>

Is the right way to use locker?

> m=ReentrantLock()  
> Threads.@threads for …  
> …  
> lock(m)  
> push!(DataFrame,Data)  
> unlock(m)  
> …  
> end

---

<div class="post-metadata">

**Author:** ![Sijun](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@Sijun](https://discourse.julialang.org/u/Sijun)\
**Post date:** [January 5, 2020, 12:18am UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949/3 "2020-01-05T00:18:40Z")

</div>

Locks (ReentrantLock, SpinLock, etc) has high overhead. So I would rather do something like:

```julia
using BenchmarkTools

@btime let
    df = DataFrame(x=1:5, y='a':'e')
    dfthr = [similar(df, 0) for i in 1:nthreads()]

    @threads for i in 1:1000
        push!(dfthr[threadid()], (rand(1:10), rand('a':'e')))
    end

    df2 = vcat(df, dfthr...)
end
# 125.201 μs (1466 allocations: 96.47 KiB)

```

Compare to :

```julia
@btime let
    df = DataFrame(x=1:5, y='a':'e')
    m = ReentrantLock()
    @threads for i in 1:1000
        lock(m)
        push!(df, (rand(1:10), rand('a':'e')))
        unlock(m)
    end
end
# 1.923 ms (6033 allocations: 148.08 KiB)

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 5, 2020, 12:29am UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949/4 "2020-01-05T00:29:09Z")

</div>

You can use Transducers.jl for this (disclaimer: I’m the author):

> [@Thread- and process-based parallelisms in Transducers.jl (+ some news)](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/1):
>
> Transducers.jl now can be used to create various table types from DataFrames.jl, TypedTables.jl, StructArrays.jl, etc. See [`copy`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.copy) and its parallel versions [`tcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.tcopy) and [`dcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dcopy).

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [January 5, 2020, 7:16pm UTC](https://discourse.julialang.org/t/would-somebody-suggest-what-is-the-right-to-work-in-a-parallel-loop-and-write-data-to-the-same-dataframe/32949/5 "2020-01-05T19:16:58Z")

</div>

Thank you so much