# Mean of multiple DataFrames

**URL:** <https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [February 10, 2021, 11:30am UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983 "2021-02-10T11:30:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sleepy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sleepy/32/20296_2.png) [@Sleepy](https://discourse.julialang.org/u/Sleepy)\
**Post date:** [February 10, 2021, 11:30am UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/1 "2021-02-10T11:30:14Z")

</div>

Hi,

we are given two or more DataFrames

```julia
using DataFrames, Statistics

df1 = DataFrame(A = 1:5, B = 3:7)
df2 = DataFrame(A = 10:14, B = 3:7)

```

What is a fast way to calculate the mean of these frames?

In the example above, the result should be also a dataframe where the entries of first column equals (1+10)/2, (2+11)/2… and the second column (3+3)/2,(4+4)/2,…

Thank you!

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 10, 2021, 12:12pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/2 "2021-02-10T12:12:47Z")

</div>

A simple way:

```julia
julia> (df1 .+ df2) ./ 2
5×2 DataFrame
 Row │ A B       
     │ Float64 Float64 
─────┼──────────────────
   1 │ 5.5 3.0 
   2 │ 6.5 4.0 
   3 │ 7.5 5.0 
   4 │ 8.5 6.0 
   5 │ 9.5 7.0

```

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 10, 2021, 12:26pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/3 "2021-02-10T12:26:28Z")

</div>

I do notice the names of the columns much match In Julia (seems sensible) and also length or you get:

ERROR: DimensionMismatch(“arrays could not be broadcast to a common size; got a dimension with lengths 5 and 4”)

so I checked in R, and it’s the same unless a multiple of the length:

> A ← 1:2  
> B ← 1:3  
> A+B  
> [1] 2 4 4  
> Warning message:  
> In A + B : longer object length is not a multiple of shorter object length  
> B ← 1:4  
> A+B  
> [1] 2 4 4 6

I was maybe expecting NA or NaN for extra rows (is there an easy way?) but more importantly any idea about what’s the idea behind the repeating/multiple R behavior and if Julia’s DataFrames should support such (maybe optionally).

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 10, 2021, 12:34pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/4 "2021-02-10T12:34:50Z")

</div>

This is called [recycling](https://www.oreilly.com/library/view/the-art-of/9781593273842/ch02s03.html) in R and imho is a terrible footgun - Julia is quite consistent in asking users to be explicit in their intentions rather than trying to guess and rely on [DWIM](https://en.wikipedia.org/wiki/DWIM), which I think is one of the strenghts of the language. It does mean more verbosity/less convenience in some situations, but I think it’s a tradeoff well worth making.

Incidentally I think this is an excellent example where the Julia behaviour makes life easier: the broadcasted dot makes it clear that addition happens elementwise, and it doesn’t make sense to do this for shapes that don’t match - rather than coming up with a “solution” to this “problem” for the user, DataFrames asks people to be explicit what they think should happen in these cases. Your `missing` suggestion would mean just pad out the smaller DataFrame with `missing` (where, btw - at the end? The start? Randomly in between?), but my guess is that in 9x% of all cases where this happens the fact that someone tries to add different-sized DataFrames is actually a bug in their code, and it’s helpful that an error is raised rather than a silent workaround performed in the background.

---

<div class="post-metadata">

**Author:** ![Sleepy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sleepy/32/20296_2.png) [@Sleepy](https://discourse.julialang.org/u/Sleepy)\
**Post date:** [February 10, 2021, 12:58pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/5 "2021-02-10T12:58:55Z")

</div>

> [@nilshg](#):
>
> `(df1 .+ df2) ./ 2`

Another question. Consider an array of DataFrames e.g. [df1,df2,df3,…,dfn].

How would you calculate the mean in this case?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 10, 2021, 1:01pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/6 "2021-02-10T13:01:21Z")

</div>

```julia

julia> reduce(.+, [df1, df2, df3]) ./ 3
5×2 DataFrame
 Row │ A B       
     │ Float64 Float64 
─────┼───────────────────
   1 │ 8.66667 3.0
   2 │ 9.66667 4.0
   3 │ 10.6667 5.0
   4 │ 11.6667 6.0
   5 │ 12.6667 7.0

```

or more generally:

```julia
reduce(.+, df_array) ./ length(df_array)

```

---

<div class="post-metadata">

**Author:** ![Sleepy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sleepy/32/20296_2.png) [@Sleepy](https://discourse.julialang.org/u/Sleepy)\
**Post date:** [February 10, 2021, 1:14pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/7 "2021-02-10T13:14:16Z")

</div>

> [@nilshg](#):
>
> `reduce(.+, [df1, df2, df3]) ./ 3`

Unfortunately, I got an error

```julia
using DataFrames, Statistics

df1 = DataFrame(A = 1:5, B = 3:7)
df2 = DataFrame(A = 10:14, B = 3:7)

reduce(.+, [df1, df2]) ./ 2

ERROR: LoadError: UndefVarError: .+ not defined

```

Using Julia Version 1.5.3 (2020-11-09)

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 10, 2021, 2:15pm UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/8 "2021-02-10T14:15:44Z")

</div>

Ah sorry, how quickly you forget… 🙂 The parsing of `.+` has actually changed in Julia 1.6 (I’m using the release candidate), see point 6 on the release notes here: [https://github.com/JuliaLang/julia/blob/release-1.6/NEWS.md#new-language-features](https://github.com/JuliaLang/julia/blob/release-1.6/NEWS.md#new-language-features)

To get something similar in 1.5 you can use an anonymous function:

```julia
julia> using DataFrames

julia> df_array = [DataFrame(A = rand(5), B = 1:5) for _ ∈ 1:10_000];

julia> reduce((x, y) -> x .+ y, df_array) ./ length(df_array)
5×2 DataFrame
 Row │ A B       
     │ Float64 Float64 
─────┼───────────────────
   1 │ 0.50288 1.0
   2 │ 0.5007 2.0
   3 │ 0.49985 3.0
   4 │ 0.499544 4.0
   5 │ 0.502726 5.0

```

---

<div class="post-metadata">

**Author:** ![ConnectedSystems](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/connectedsystems/32/16815_2.png) [@ConnectedSystems](https://discourse.julialang.org/u/ConnectedSystems)\
**Post date:** [April 17, 2022, 1:21am UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/9 "2022-04-17T01:21:29Z")

</div>

Here’s an alternative which may be useful.  
The advantage is that any stat function could be applied

```julia
using Statistics

# Create a DF to store results into
# There are more efficient ways of doing this but here I just copy one of the existing DFs
mean_df = copy(df1)  

# Calculate the mean
mean_df[:, :] = mean(cat(map(Matrix, [df1, df2, df3]), dims=3))

```

Explanation:

The call to `map()` converts each DataFrame into a matrix.

`cat(..., dims=3)` then concatenates these along the third dimension, creating an N-dimensional matrix

We can then simply apply the stat function (`mean()` in this case, but could be any function) to the result.

`mean_df[:, :] =` assigns the values in the target DataFrame (note the `[:, :]`)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [April 17, 2022, 7:58am UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/10 "2022-04-17T07:58:51Z")

</div>

Nice.  
An alternative to the 3rd dimension could be to [use this tip](https://discourse.julialang.org/t/broadcasting-with-vectors-of-static-vectors/12693/2) to use a tuple shield to broadcast over the array of matrices:

```julia
using DataFrames, Statistics
dfn = [DataFrame(i*ones(4,3),:auto) for i in 1:4] 
m₀, = mean.((map(Matrix, dfn),))
σₘ, = std.((map(Matrix, dfn),))

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [April 17, 2022, 9:12am UTC](https://discourse.julialang.org/t/mean-of-multiple-dataframes/54983/11 "2022-04-17T09:12:43Z")

</div>

I am one year late to the party. Let me add that in Julia broadcasting has one special case when recycling is allowed. It is when one of the collections has length 1 in some dimension. Example:

```julia
julia> [1 2] .+ [11 12; 13 14]
2×2 Matrix{Int64}:
 12 14
 14 16

```

and the same applies to `DataFrames`:

```julia
julia> df1 = DataFrame([1 2], :auto)
1×2 DataFrame
 Row │ x1 x2
     │ Int64 Int64
─────┼──────────────
   1 │ 1 2

julia> df2 = DataFrame([11 12; 13 14], :auto)
2×2 DataFrame
 Row │ x1 x2
     │ Int64 Int64
─────┼──────────────
   1 │ 11 12
   2 │ 13 14

julia> df1 .+ df2
2×2 DataFrame
 Row │ x1 x2
     │ Int64 Int64
─────┼──────────────
   1 │ 12 14
   2 │ 14 16

```