Hi there. Does anyone know a function or a package that can do the same DataFrame groupby function on a regular array? e.g. julia> A = [1,2,3,4,1,2,4,4] 8-element Array{Int64,1}: 1 2 3 4 1 2 4 4 Then, sortperm gets me half way: julia> sortperm(A) 8-element Array{Int64,1}: 1 5 2 6 3 …

Never mind. There is already a good post. [image] Allocation - groupby Performance I’m attempting to create a function groupby(f, itr) that returns a dictionary whose keys are the results of applying f to the elements of itr, and the values are some collection of the ele…

Query.jl works on regular arrays, so you can just use its @groupby command.

Interesting. I’ve tried but unsure how to get the indices rather than values. julia> A = [1,2,3,4,1,2,4,4] 8-element Array{Int64,1}: 1 2 3 4 1 2 4 4 julia> A |> @groupby(_) ?-element query result [1, 1] [2, 2] [3] [4, 4, 4]

I got it! julia> 1:length(A) |> @groupby(A[_]) |> collect 4-element Array{QueryOperators.Grouping{Any,Int64},1}: [1, 5] [2, 6] [3] [4, 7, 8]

I was curious about performance. Since it’s not a tough problem, I’ve created a custom groupby function and compared it with Query.jl. Just sharing results: https://gist.github.com/tk3369/f979a4292fd696bee753f37cae93b45c

function sampleRanks of package NormalizeQuantiles.jl maybe also a solution. Just for your information. [image] GitHub - oheil/NormalizeQuantiles.jl: NormalizeQuantiles.jl implements... NormalizeQuantiles.jl implements quantile normalization - GitHub - oheil/NormalizeQuantiles.jl: Normal…

julia> Pkg.add("NormalizeQuantiles") julia> using NormalizeQuantiles julia> A = [1,2,3,4,1,2,4,4]; julia> (r,m) = sampleRanks(A,resultMatrix=true); julia> m Dict{Int64,Array{Int64,N} where N} with 4 entries: 4 => [4, 7, 8] 2 => [2, 6] 3 => [3] 1 => [1, 5]

julia> @time 1:length(A) |> @groupby(A[_]) |> collect 0.027596 seconds (9.28 k allocations: 522.957 KiB) julia> @time A |> @groupby(_) 0.004368 seconds (1.95 k allocations: 118.646 KiB) julia> @time (r,m) = sampleRanks(A,resultMatrix=true) 0.000128 seconds (234 allocations: 11.938 KiB)

Need to use BenchmarkTools for proper benchmarking. Tried NormalizedQuantiles. It returns a Dict, which is nice, but it’s much slower than others and it does not scale with larger arrays. See updated gist for benchmark details. Method Function 8 elements 1,000 elements 10,000 elements Cu…

Your custom function is quite fast. I think the main performance loss is because I have to expect dirty data, e.g. A = [1,2,NaN,3,4,1,“5,0”,5.0,2,4,4] where anything which is not of type “a number in general” is NA (not available, missing, …, like NA in R) However, while comparing with Query.jl a…

Groupby for regular array

General Usage

tk3369 March 22, 2018, 6:04am 2

Never mind. There is already a good post.

Topic		Replies	Views
Various by-group strategies compared Data	36	4167	January 30, 2018
Groupby function? General Usage	7	5137	March 23, 2018
Groupby() on two individual arrays General Usage question , array	1	349	April 20, 2021
What functions/packages should I use to sort and "group by" as fast as possible...? Performance sort , dataframes	4	1494	December 16, 2018
Groupby on an expression or a vector? New to Julia	21	764	June 11, 2024

Groupby for regular array

Related topics