Concatenating DataFrames in parallel

I am trying to run multiple simulations in parallel. The output of these simulations is a DataFrame.
My goal is to return a concatenated DataFrame with all of the simulations.

The simple code would be

using DataFrames
df_res = DataFrame()
for i in 1:3
  df_res = append!(df_res, DataFrame(a = i, b = i^2))

I am not sure how to make this happen using multiple threads.
I have found that I could use transducers as follow

using Transducers, BangBang
df_res = foldxt(append!!, 
    Map(x -> DataFrame(a=x, b=x^2)),
    1:3; init = Empty(DataFrame))

I am not sure that this is an efficient way of running things in parallel or whether there is an idiomatic of doing what I am trying to do.

Surprisingly, I found a simple reduce(vcat, dfvec) works pretty dang good because there is a fastpath for this when the input is a vector of DataFrames. So, in your case, I would run the simulations in parallel and then use that once at the end.

This is the intended way to do it. Although it is single threaded it will be fast. The benefit from parallelizing it would be minimal.

Just to be clear you recommend that I do something like:

pmap(x -> DataFrame(a = x, b = x^2), 1:3) |> (x -> reduce(vcat, x))

Yes, 2 steps. First simulate and get a vector of DataFrames, then pass that to reduce (although I wouldn’t write it like you have).

Also for the parallel simulations, look into as a possible alternative to pmap, depending on the size of your problem.