As shown by the code below, Julia is twice slower than R (!) to sum elements of a Float64 vector with missing values, even with Julia 0.7. In R, missing values are encoded with sentinelle values. In Julia, they are encoded with different type. This difference of design makes it much slower to mani…

See the two last sections of my blog post . We get an incredibly high performance on recent git master for a manual sum over Vector{Int}, a similar performance as R for a manual sum over Vector{Float64}, but a somewhat poorer performance using skipmissing. I’ve filed this issue about it yesterday. A…

Don’t benchmark in global scope ( performance tips ) Use BenchmarkTools for benchmarking The first run includes compile time

The issue here is likely using Missing. If you have to do that, then you’re not on Julia v0.7-alpha. The optimizations to make this fast only exist on the newest alpha version.

Using BenchmarkTools (as suggested by Stefan) and Julia v0.7-alpha (as suggested by Chris), I see Julia being only slightly slower than R: Edit: And the very latest Julia nightlies are even faster than R (see below) julia> using BenchmarkTools julia> @btime sum(skipmissing($y)) 1.329 s (6 alloc…

Why is your sum implementation so much faster than the one with skipmissing though? What’s going on in the skipmissing part that’s slowing it down? I assumed it was just an iterator that skipped over the missing values which should be almost equivalent to what you have?

Yeah, I was surprised by that as well. It also allocates some memory, even if I omit the skipmissing() itself: julia> @btime sum($(skipmissing(y))) 1.323 s (5 allocations: 80 bytes) It looks like the implementation of SkipMissing iteration hasn’t been updated to use the new iterate() protocol as…

Holy cow! Yeah, it’s 4 times faster (and more than 2X faster than R) on the very latest Julia nightly: julia> versioninfo() Julia Version 0.7.0-alpha.214 Commit ffc9182b3a (2018-06-20 19:45 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Core(TM) i7-3920XM CPU @ 2.90GHz WORD…

Hi, This does not change anything. I obtain these results in v0.7, even with BenchmarkTools and using local scope. See below. To be more precise, I obtain 0.5s in Julia vs 0.2s in R using BenchmarkTools, Compat x = rand(200_000_000) y = replace(x-> x<= 0.01, x, missing) function f(y) @btime sum(…

y is still a non-const global variable. See https://docs.julialang.org/en/stable/manual/performance-tips/#Avoid-global-variables-1 and compare to how rdeits is doing the benchmarking in With Missings, Julia is slower than R - #4 by rdeits .

I thought that your use of zero(eltype(x)) would be problematic, but apparently zero(Union{Missing, T}) is equivalent to zero(T). What is the general rule governing this? (Edit: promote_type, maybe?)

The general rule is that missing behaves as if it were some value in the domain T but you don’t know which one. So zero(Union{Missing, T}) = zero(T) makes sense since the zero of a missing T value is just the zero of T.

With Missings, Julia is slower than R

General Usage

StefanKarpinski June 21, 2018, 3:19am 2

Don’t benchmark in global scope (performance tips)
Use BenchmarkTools for benchmarking
The first run includes compile time

CSV Reading (rewrite in C?)

Topic		Replies	Views
Missing or NaN General Usage	26	12635	August 1, 2018
Why I love Julia Community announcement	34	5047	October 11, 2019
Why are missing values not ignored by default? Internals & Design data , missing-values	330	10607	January 17, 2024
Fastest way to count unique elements in `Vector{Union{Bool, Missing})` Performance	35	7094	January 16, 2018
Micro-benchmark General Usage	7	800	October 5, 2023

With Missings, Julia is slower than R

Related topics