# Who does "better" than DataFrames?

**URL:** <https://discourse.julialang.org/t/who-does-better-than-dataframes/96946>\
**Category:** Performance\
**Tags:** dataframes\
**Created:** [April 1, 2023, 1:12pm UTC](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946 "2023-04-01T13:12:18Z")\
**Posts on this page:** 4\
**Page:** 3

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 4, 2023, 8:26pm UTC](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946/41 "2023-04-04T20:26:09Z")

</div>

Maybe I’m missing something but, at first accept, I would say that this solution is based on the specific situation that the dataframe is already sorted by groups.  
The test should be done in the case of a “shuffled” dataframe to get a more meaningful answer.

```julia
using Random
s=shuffle(repeat(1:10^6, inner=4))
df=DataFrame(;s,t,r)
@assert combine(groupby(df, :s),:r=>maximum).r_maximum == vec(maximum(reshape(r, 4, :), dims=1))
ERROR: AssertionError: (combine(groupby(df, :s), :r => maximum)).r_maximum == vec(maximum(reshape(r, 4, :), dims = 1))
Stacktrace:
 [1] top-level scope
   @ c:\Users\sprmn\.julia\environments\v1.8.3\dataframes33.jl:414    

```

edit  
Now I read better the premise of the post, which specifies the scope of validity of the proposed solution.

> [@JackDevine](#):
>
> In this particular case you can use the structure of the `s` index in `df` to beat DataFrames:

in any case, the proposal, adapted to the general situation, does not disfigure at all.

```julia
julia> @btime begin
       sort!(df,:s)
       maximum(reshape(df.r, 4, :), dims=1)
       end
  30.534 ms (22 allocations: 114.44 MiB)
1×1000000 Matrix{Float64}:
 0.742695 0.952913 0.884315 … 0.771818 0.929275 0.943746

julia> @btime combine(groupby(df, :s),:r=>maximum)
  32.264 ms (348 allocations: 55.33 MiB)
1000000×2 DataFrame

```

a variant that makes better use of the particularity

```julia
julia> @btime begin
       sort!(df,[:s,:r])
       reshape(df.r, 4, :)[4,:]
       end
  25.522 ms (45 allocations: 114.44 MiB)
1000000-element Vector{Float64}:

```

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [April 4, 2023, 11:40pm UTC](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946/42 "2023-04-04T23:40:13Z")

</div>

My point is, regarding

> It’s a shame these optimizations are only available for a single type of a single package.

that I don’t think it’s a big deal if it only works for one type or not, since you can always wrap your types in another to access the functionality (and that doesn’t need to cost in performance). I don’t really agree re- “flat tables” vs base types; a DataFrame is just a collection of vectors and doesn’t imply the contents must be “flat”. Agreed that StructTypes provides very similar functionality.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 5, 2023, 7:59am UTC](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946/43 "2023-04-05T07:59:05Z")

</div>

> [@ericphanson](#):
>
> that I don’t think it’s a big deal if it only works for one type or not, since you can always wrap your types in another to access the functionality

Well, for me it seems pretty weird if for performance instead of simply `unique(s)` users need to install DataFrames, then do:

```julia
using DataFrames
df = DataFrame(;s)
combine(groupby(df, :s), :s => first).s_first

```

(example from slightly above in this thread).

> [@ericphanson](#):
>
> I don’t really agree re- “flat tables” vs base types; a DataFrame is just a collection of vectors and doesn’t imply the contents must be “flat”.

Yeah, it can store any data type in columns, but common dataframe operations only work at the “top level” – on the level of columns as a whole. Sure, you can store the dataset as-is in one of the columns, but we are hardly speaking about convenience anymore. And this only works if your data is a vector, not an n-dim array. Additionally, many functions take real collections in Julia, and dataframes don’t follow the collection interface.

Anyway, not sure what exactly is your main point. I just said it would be great if these performance optimizations were available for many other julia container types. Sure, it’s lots of work for identifying common, generally useful blocks and using them to optimize even Base functions. So, this may even never be done…

---

<div class="post-metadata">

**Author:** ![JackDevine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jackdevine/32/1048_2.png) [@JackDevine](https://discourse.julialang.org/u/JackDevine)\
**Post date:** [April 6, 2023, 12:55am UTC](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946/44 "2023-04-06T00:55:25Z")

</div>

> [@rocco\_sprmnt21](#):
>
> in any case, the proposal, adapted to the general situation, does not disfigure at all.

Yes, that does make it slightly more general, but it still isn’t entirely general since it fails if one of the groups doesn’t have 4 elements. So you have to be careful when using that strategy and you would probably want to add assertions/checks to your code to make sure that problematic data doesn’t slip through.

> [@aplavin](#):
>
> Well, for me it seems pretty weird if for performance instead of simply `unique(s)` users need to

I agree and I think that the optimizations should be added to Base. Although first, it would be nice to try lots of different scenarios (random data, string data, data where checking equality is expensive) to see if the performance improvement is uniform.

Also, kudos to the DataFrames.jl developers for beating Base for this particular task!

[Previous page](https://discourse.julialang.org/t/who-does-better-than-dataframes/96946.md?page=2)