@jangorecki has released the newest Database-like ops benchmark (by the way - thank you Jan for jour fantastic work!). I have decided to write this post as for the first time with DataFrames.jl 0.21.4 + CSV.jl 0.7.1 we have managed to successfully pass all groupby tests (and it is really selective, …

Also Dataframes 1.3 and Julia 1.7 will speed things up a bunch. Unfortunately, it seems mostly unmaintained currently.

[image] xinchin: DataFrames.jl only use 20 out of 40! This is a tricky decision. Using more logical cores than there is physical cores does not have to lead to performance boost. Such things would need to be benchmarked, but as @Oscar_Smith noted - the benchmark is currently not maintained. H…

I think the maintainer wouldn’t like to spend too much time on it since data.table is not that much impressive as it used to be few years ago. :thinking:

That’s uncharitable. They were really good benchmarks. People just get busy.

Seems like DuckDB has resurrected the H2O benchmark and plans to keep re-running it: The Return of the H2O.ai Database-like Ops Benchmark - DuckDB

[image] bkamins: My ideal solution would be to have a general package that would perform row matching for joins getting some type stable input, and in DataFrames.jl we would just delegate this part of work to it and would concentrate only on composing the result. @bkamins any chance this effo…

FlexiJoins.jl is the general type-stable joining package. It doesn’t constrain the dataset type, and is based on the regular Julia collections interface (not on Tables.jl, but many tables are implicitly supported). The generic code path is implemented to be reasonably fast, but it doesn’t have all …

Interestingly both DuckDB and Polars use Arrow column-oriented storage.

The state of DataFrames.jl H2O benchmark

Package Announcements

Juan June 4, 2022, 12:54am 40

The new DLMReader packages is supposed to improve the benchmarks: [ANN] DLMReader: the most versatile Julia package for reading delimited files yet - #13 by sl-solution

Topic		Replies	Views
Julia performs poorly on group-by benchmarks Data performance	48	6045	January 23, 2019
How is the data ecosystem right now for large datasets? Data	35	6984	July 13, 2017
[ANN] DataFrameDBs.jl Data package , announcement	60	4379	May 2, 2020
DataFrames.jl data engineering performance compared with other softwares Performance performance	6	1052	November 10, 2021
Who does "better" than DataFrames? Performance dataframes	43	2292	April 6, 2023

The state of DataFrames.jl H2O benchmark

Related topics