Concatenating DataFrames in parallel

Specific Domains Data
1

I am trying to run multiple simulations in parallel. The output of these simulations is a DataFrame.
My goal is to return a concatenated DataFrame with all of the simulations.

The simple code would be

using DataFrames
df_res = DataFrame()
for i in 1:3
  df_res = append!(df_res, DataFrame(a = i, b = i^2))
end

I am not sure how to make this happen using multiple threads.
I have found that I could use transducers as follow

using Transducers, BangBang
df_res = foldxt(append!!, 
    Map(x -> DataFrame(a=x, b=x^2)),
    1:3; init = Empty(DataFrame))

I am not sure that this is an efficient way of running things in parallel or whether there is an idiomatic of doing what I am trying to do.

2

Surprisingly, I found a simple reduce(vcat, dfvec) works pretty dang good because there is a fastpath for this when the input is a vector of DataFrames. So, in your case, I would run the simulations in parallel and then use that once at the end.

3

This is the intended way to do it. Although it is single threaded it will be fast. The benefit from parallelizing it would be minimal.

4

Just to be clear you recommend that I do something like:

pmap(x -> DataFrame(a = x, b = x^2), 1:3) |> (x -> reduce(vcat, x))
5

Yes, 2 steps. First simulate and get a vector of DataFrames, then pass that to reduce (although I wouldn’t write it like you have).

Also for the parallel simulations, look into ThreadsX.map as a possible alternative to pmap, depending on the size of your problem.