# Understanding DataFrame allocations

**URL:** <https://discourse.julialang.org/t/understanding-dataframe-allocations/122792>\
**Category:** Performance\
**Tags:** dataframes\
**Created:** [November 18, 2024, 10:04pm UTC](https://discourse.julialang.org/t/understanding-dataframe-allocations/122792 "2024-11-18T22:04:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![miguelborrero](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@miguelborrero](https://discourse.julialang.org/u/miguelborrero)\
**Post date:** [November 18, 2024, 10:04pm UTC](https://discourse.julialang.org/t/understanding-dataframe-allocations/122792/1 "2024-11-18T22:04:34Z")

</div>

Hi there,

The split-apply-combine strategy is something that I use a lot so Im trying to understand a bit more what goes under the hood so that I can write better code. I was just looking at the following example:

```julia
df = DataFrame(x = rand(20), y = rand(20))
function test(df)
           df |>
           x -> transform!(x, :x => ByRow(val -> 2*val) => identity)
end 
function test2(df)
           df.x = 2 .* df.x
end

```

`@btime` on `test` gives  
 ![Screenshot 2024-11-18 at 14.01.09](https://global.discourse-cdn.com/julialang/original/3X/2/d/2d2ab8bc88a9c9884cc3149f5df5423851779361.png)

While `@btime` on `test2` gives  
 ![Screenshot 2024-11-18 at 14.01.36](https://global.discourse-cdn.com/julialang/original/3X/b/1/b19c832a5a4b194233734c73e90184010be87fd2.png)

My first question is: where are the 3 allocations coming from in the first case and the more important second question: why is there such a huge difference between the piping + transform! implementation? I though it was an in-place method.

Thanks!

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 18, 2024, 10:16pm UTC](https://discourse.julialang.org/t/understanding-dataframe-allocations/122792/2 "2024-11-18T22:16:22Z")

</div>

The infrastructure for dealing with the `src => fun => dest` inputs can be somewhat complicated, and this leads to allocations.

Fortuntaely, this is a fixed cost. When a data frame gets bigger, the difference between the two functions disappears

```julia
julia> df = DataFrame(x = rand(1_000_000), y = rand(1_000_000));

julia> @btime test($df);
  936.341 μs (92 allocations: 7.63 MiB)

julia> @btime test2($df);
  928.734 μs (4 allocations: 7.63 MiB)

```