@jangorecki has released the newest Database-like ops benchmark (by the way - thank you Jan for jour fantastic work!). I have decided to write this post as for the first time with DataFrames.jl 0.21.4 + CSV.jl 0.7.1 we have managed to successfully pass all groupby tests (and it is really selective, for 50GB size; e.g. pandas or dplyr do not pass it). Passing this last hurdle was possible mainly due to work of @quinnj so :clap: (of course many others have worked hard for years to make this happen). What are the key take-aways for groupby tests: if we ignore compilation time, we are roughly on par with the fastest options if number of groups is not very large; if there are many groups the results are mixed, but there are cases when we are very bad; @nalimilan has been recently working on improving it (and in particular taking advantage of multi threading) - so I am anxiously waiting to see what he came up with; when pandas or dplyr work (these I would say are typical packages regular users work with) we are either on par or significantly better What we learn from join tests: we are very, very bad here (in terms of both: performance and memory usage) and this is the key think we should focus on improving (we knew it, but it is just confirmed again)

Also Dataframes 1.3 and Julia 1.7 will speed things up a bunch. Unfortunately, it seems mostly unmaintained currently.

[image] xinchin: DataFrames.jl only use 20 out of 40! This is a tricky decision. Using more logical cores than there is physical cores does not have to lead to performance boost. Such things would need to be benchmarked, but as @Oscar_Smith noted - the benchmark is currently not maintained. Hopefully it would change in the future.

I think the maintainer wouldn’t like to spend too much time on it since data.table is not that much impressive as it used to be few years ago. :thinking:

That’s uncharitable. They were really good benchmarks. People just get busy.

Seems like DuckDB has resurrected the H2O benchmark and plans to keep re-running it: The Return of the H2O.ai Database-like Ops Benchmark - DuckDB

[image] bkamins: My ideal solution would be to have a general package that would perform row matching for joins getting some type stable input, and in DataFrames.jl we would just delegate this part of work to it and would concentrate only on composing the result. @bkamins any chance this effort could be implemented for general Tables.jl? It would be super nice to reuse the work in other table types and contexts.

FlexiJoins.jl is the general type-stable joining package. It doesn’t constrain the dataset type, and is based on the regular Julia collections interface (not on Tables.jl, but many tables are implicitly supported). The generic code path is implemented to be reasonably fast, but it doesn’t have all those special cases DataFrames.jl do, such as integer or low-cardinality optimizations.

Interestingly both DuckDB and Polars use Arrow column-oriented storage.

The state of DataFrames.jl H2O benchmark

Package Announcements

Juan June 4, 2022, 12:54am 40

The new DLMReader packages is supposed to improve the benchmarks: [ANN] DLMReader: the most versatile Julia package for reading delimited files yet - #13 by sl-solution

Topic		Replies	Views
Julia performs poorly on group-by benchmarks Data performance	48	6669	January 23, 2019
How is the data ecosystem right now for large datasets? Data	35	7351	July 13, 2017
[ANN] DataFrameDBs.jl Data package , announcement	60	5172	May 2, 2020
DataFrames.jl data engineering performance compared with other softwares Performance performance	6	1170	November 10, 2021
Who does "better" than DataFrames? Performance dataframes	43	2822	April 6, 2023

The state of DataFrames.jl H2O benchmark

Related topics