# Efficient computation of statistics across multiple data frames

**URL:** <https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [January 30, 2024, 8:18pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471 "2024-01-30T20:18:13Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [January 30, 2024, 8:18pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/1 "2024-01-30T20:18:13Z")

</div>

I have a vector collection of DataFrames that have the same number of rows and same column names and I would like to compute statistics across all data frames to arrive at a data frame that is the same size as a single data frame in the original collection. For instance how can I most efficiently calculate the mean for each row of c1 and the median for each row of c2 across all data frames given the below example:

```julia
using DataFrames

n = 10
dfs =[]
for i = 1:n
    dfs = push!(dfs, DataFrame(name=["a","b","c","d","e"], c1=rand(5), c2=rand(5)))
end

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 30, 2024, 8:25pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/2 "2024-01-30T20:25:49Z")

</div>

Just so we are clear, do you want `mean(df.c1)` separately for each value of `name` in the data frame, or do you want a mean for each row of the variables `:c1` and `:c2`?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [January 30, 2024, 8:25pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/3 "2024-01-30T20:25:58Z")

</div>

Easiest way might be to make a vector of vectors out of the columns from all dfs you want to work on, run the aggregation on that. This way it’s at least type stable.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 30, 2024, 8:31pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/4 "2024-01-30T20:31:00Z")

</div>

I think OP probably wants (using DataFramesMeta)

```julia
for i in 1:n
    @transform! dfs[n] begin 
        :c1_mean = mean(:c1)
        :c2_med = median(:c2)
    end
end

```

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [January 30, 2024, 8:50pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/5 "2024-01-30T20:50:07Z")

</div>

> [@pdeffebach](#):
>
> just so we are clear, do you want `mean(df.c1)` separately for each value of `name` in the data frame, or do you want a mean for each row of the variables `:c1` and `:c2`?

I’d like the mean for each row of `c1` across all data frames. The result will be a single data frame with 5 rows.

the brute force way would look like this:

```julia
mean(hcat(dfs[1].c1, dfs[2].c1, dfs[3].c1, dfs[4].c1, dfs[5].c1, dfs[6].c1, dfs[7].c1, dfs[8].c1, dfs[9].c1, dfs[10].c1), dims = 2)

```

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [January 30, 2024, 10:23pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/6 "2024-01-30T22:23:59Z")

</div>

In this case, I would just try to get all my data into one data frame and then use grouped aggregations:

```julia
df_long = reduce(vcat, dfs)
# or if you want to keep track where each datum came from
df_long = reduce(vcat, transform(df, [] => (() -> i) => :id) for (i, df) in enumerate(dfs))

combine(groupby(df_long, :name),
        :c1 => mean,
        :c2 => median)

```

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [January 30, 2024, 11:45pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/7 "2024-01-30T23:45:48Z")

</div>

This is exactly what I was looking for!

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [January 31, 2024, 5:19am UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/8 "2024-01-31T05:19:00Z")

</div>

The `vcat(dfs...)` method for DataFrames also has a special `source` keyword arg which does this for you.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [January 31, 2024, 4:27pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/9 "2024-01-31T16:27:56Z")

</div>

This, although not very elegant, is quite efficient

```julia
m3=hvncat(3,Matrix.(dfs)...)[:,[2,3],:]
mc1=mean(m3[:,1:1,:],dims=3)[:]
mdc2=median(m3[:,2:2,:],dims=3)[:]
name='a':'e'
DataFrame((;name,mc1,mdc2))

```

PS  
Furthermore there is the use of a little used function and with documentation that should be enriched with examples

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [January 31, 2024, 6:55pm UTC](https://discourse.julialang.org/t/efficient-computation-of-statistics-across-multiple-data-frames/109471/10 "2024-01-31T18:55:54Z")

</div>

Nice, thanks.