As shown by the code below, Julia is twice slower than R (!) to sum elements of a Float64 vector with missing values, even with Julia 0.7. In R, missing values are encoded with sentinelle values. In Julia, they are encoded with different type. This difference of design makes it much slower to mani…

See the two last sections of my blog post . We get an incredibly high performance on recent git master for a manual sum over Vector{Int}, a similar performance as R for a manual sum over Vector{Float64}, but a somewhat poorer performance using skipmissing. I’ve filed this issue about it yesterday. A…

The general rule is that missing behaves as if it were some value in the domain T but you don’t know which one. So zero(Union{Missing, T}) = zero(T) makes sense since the zero of a missing T value is just the zero of T.

Thanks to nalimilan for being on this already! Same issue happens with sort though. Julia takes 1.5 R to sort a Float64 vector without missing values, but this jumps to 3.5 R to sort a Float64 vector with missing values. Do you think there will be a way to make sorting roughly as fast as R? For @r…

Can you post (a) the Julia and R code you’re using, (b) the actual results, and (c) the exact versions of both? Without that information it’s impossible to say anything at all.

# Julia Version 0.7.0-beta.9 Y1 = rand(Float64, 10_000_000); Y2 = ifelse.(rand(length(Y1)) .< 0.9, Y1, missing); @btime sort($Y1) #> 764.978 ms (2 allocations: 76.29 MiB) @btime sort($Y2) #> 1.797 s (4 allocations: 128.75 MiB) # in R version 3.5.0 (2018-04-23) Y1 = rnorm(10000000) Y2 = ifelse(…

I get essentially no difference between Julia v0.7.0-beta.5 and R 3.4.4 when sorting Float64s on my 2018 i7 laptop: julia> @btime sort(y) setup=(y = rand(Float64, 10_000_000)) 906.928 ms (2 allocations: 76.29 MiB) (I’m putting the generation of y into the setup to also randomize the initial sort…

Good catch! Looking at the time profile and at allocations, I couldn’t find anything obvious. It would be worth filing an issue. Keep up testing performance! It’s hard to think about all possible tests.

Is it possible that it’s not using the floating-point quicksort code and is using a merge sort instead?

[image] StefanKarpinski: Is it possible that it’s not using the floating-point quicksort code and is using a merge sort instead? Seems like it. julia> Base.Sort.defalg(y2) Base.Sort.MergeSortAlg() julia> @btime sort(y2) setup=(y = rand(Float64, 10_000_000); y2 = ifelse.(rand(length(y)) .< 0…

Note that until the fixes are merged, a relatively straightforward solution is to use sort!(collect(skipmissing(Y2)), alg=QuickSort). It’s about as fast as R and twice as fast as sort(Y2). It’s also what R does by default (na.last=NA), i.e. skipping NAs (keeping NAs doesn’t affect performance). We …

With Missings, Julia is slower than R

General Usage

rdeits June 25, 2018, 9:34pm 19

x-ref https://github.com/JuliaLang/julia/issues/27781

1 Like

Topic		Replies	Views
Missing or NaN General Usage	26	12603	August 1, 2018
Why I love Julia Community announcement	34	5035	October 11, 2019
Why are missing values not ignored by default? Internals & Design data , missing-values	330	10518	January 17, 2024
Fastest way to count unique elements in `Vector{Union{Bool, Missing})` Performance	35	7080	January 16, 2018
Micro-benchmark General Usage	7	795	October 5, 2023

With Missings, Julia is slower than R

Related topics