What functions/packages should I use to sort and "group by" as fast as possible...?

[Juan]

I’m looking for fast functions to group by or sort large dataframes with several columns of strings (with few different values) or numbers.

There are many packages and options: Base, ShortStrings, SortingLab, FastGroupBy, SortingAlgorithms…

I have only been able to install and use ShortStrings and SortingAlgorithms, the others produce errors.

Some packages say their functionality will be merged onto base Julia o dataframe.

What do yo suggest to use?
Any special function from those packages? Or just keep with the defaults on dataframe or alternatives such as IndexedTables or CategoricalArrays?

There are other threads about this:

Julia performs poorly on group-by benchmarks

h20 has published a set of benchmarks that shows Julia’s DataFrames.jl has, in general, the worst group-by performance out of many data packages. JuliaDB.jl was not benchmarked so that may be a good addition. I have done some work before on optimising some benchmarks and I’ve been putting it off until the release of v1.0. Now that v1.0.1 is out, it’s time for me to pick up the work again using FastGroupBy.jl.

Progress towards faster `sortperm` for Strings

We kind of do this already since CSV.jl creates CategoricalArray vectors for columns with a small proportion of unique values. I think it makes sense to use CategoricalArray/PooledArray to represent values with lots of duplicates, and arrays of strings for “real” strings which are almost all different from one another.

Group-by performance benchmarks and recommendations

I have been trying to improve Julia’s DataFrame group-by for a while now and I think I am able to synthesized my thinking into APIs. Here are my synthesized recommendation (as of 25th Feb 2018) Recommendation Why? Benchmark Status If you need to perform group-by A LOT use JuliaDB.jl/IndexedTables.jl You can create indexes on the group-by columns and it is generally faster than other available methods No benchmarks; as the benchmarks was intended to test non-indexing performance If th…

but I don’t want to be told again I’m writing on old threads, even if they speak about the same.

[xiaodai]

Most of them are mt posts. I am learning how to make the package work in v1, but my package can only handle one column sorts well and multi-column sorts would need some work. Basically to make it run fast for stringa you need to radixsort the strings and in the sort one should return the sortperm as well which ca n be used to sort the other columns. This is the fastest way I found

[Juan]

But do you suggest keeping the data on frameworks and use some package to perform the radix sort? What package (ShortStrings, SortingLab, FastGroupBy, SortingAlgorithms,…)?
Or should we use other framework such as Indexedtables, Categoricalarrays, Staticarrays… instead?

[xiaodai]

The third post in your original post remains the most up to date advice. I think just use DataFramesMeta.jl or just the functions in DataFrames.jl directly. They are not fast yet, but they remain the best hope because they might get updated with faster algorithms. Ok. I will devote the next week to work on getting SortingLab.jl and FastGroupBy.jl ready for Julia v1.

[Juan]

While looking at differet benchmarks I’ve seen other promising solutions such as Dask, SciDB and Mapd.