# Any way to speed up sorting a dataframe? A more efficient sortperm would be great
**URL:** https://discourse.julialang.org/t/any-way-to-speed-up-sorting-a-dataframe-a-more-efficient-sortperm-would-be-great/41995
**Category:** General Usage
**Tags:** sort, sortperm, dataframes
**Created:** [June 24, 2020, 4:01pm UTC](https://discourse.julialang.org/t/any-way-to-speed-up-sorting-a-dataframe-a-more-efficient-sortperm-would-be-great/41995 "2020-06-24T16:01:51Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 24, 2020, 4:01pm UTC](https://discourse.julialang.org/t/any-way-to-speed-up-sorting-a-dataframe-a-more-efficient-sortperm-would-be-great/41995/1 "2020-06-24T16:01:51Z")
Consideirng a DataFrame
```julia
using DataFrames
using Random: randstring
M = 100_000_000
str_base = [randstring(8) for i in 1:1_000_000]
df = DataFrame(int = rand(Int32, M), float=rand(M), str = rand(str_base, M))
@time sort!(df, :int);
# 80s on my machine
@time sort!(df, :str);
# 170son my machine
using CSV
CSV.write("tmp.csv", df)
```
The same operation using R’s data.table is like 3s
```r
library(data.table)
df = fread("tmp.csv")
setkey(df, "int")
# 3s
setkey(df, "str")
# 25s
```
So based on this the performance of data.table is still much better.
Now the `sort!` algorithm is really simple which I can replicate here
```julia
]add https://github.com/xiaodaigh/SortingLab.jl
using SortingLab
using Base.Threads: @spawn
function another_sort!(df, col)
@time ordering = fsortperm(df[!, col])
channel_lock = Channel{Bool}(length(names(df)))
for c in names(df)
@spawn begin
v = df[!, c]
@inbounds v = v[ordering]
put!(channel_lock, true)
end
end
for _ in names(df)
take!(channel_lock)
end
df
end
@time another_sort!(df, :int); # sortperm is 10s total 12s~18s
@time another_sort!(df, :str); # sortperm is 10s total 12s~18s
```
You can see that `(f)sortperm` takes 10s. So using a more optimise `sortperm` like `SortingLab.fsortperm` can get much better results already.
The solution seems to be about finding a more efficient `sortperm`. For a start, perhaps adapting `SortingLab.fsortperm` would be a good start.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [June 25, 2020, 9:12am UTC](https://discourse.julialang.org/t/any-way-to-speed-up-sorting-a-dataframe-a-more-efficient-sortperm-would-be-great/41995/2 "2020-06-25T09:12:20Z")
> [@xiaodai](#):
>
> The solution seems to be about finding a more efficient `sortperm` . For a start, perhaps adapting `SortingLab.fsortperm` would be a good start.
Since this is suggesting a very specific improvement to a package, opening a pull request or at least an issue there might be the best way to start a discussion about this.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 25, 2020, 9:55am UTC](https://discourse.julialang.org/t/any-way-to-speed-up-sorting-a-dataframe-a-more-efficient-sortperm-would-be-great/41995/3 "2020-06-25T09:55:46Z")
Already discussed on slack.