# \`Nulls.skip\` is very slow

**URL:** <https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351>\
**Category:** Data\
**Created:** [October 9, 2017, 4:43pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351 "2017-10-09T16:43:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 9, 2017, 4:43pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351/1 "2017-10-09T16:43:33Z")

</div>

Please excuse how rough this example is (but I think it is essentially correct)

```julia
using Nulls
using BenchmarkTools
      
const N = 10^7
const n = 10^5

A = convert(Vector{Union{Int64,Null}}, rand(Int64, N))
                               
A[rand(1:N, n)] = null
    
x = 0

@btime for a ∈ $A
    !isnull(a) && ($x += a)
end

x = 0

@btime for a ∈ Nulls.skip($A)
    $x += a
end

```

this gives

```julia
  1.163 s (39602147 allocations: 604.28 MiB)
  11.625 s (148507554 allocations: 3.25 GiB)

```

Thing is, the [source code for Nulls.skip](https://github.com/JuliaData/Nulls.jl/blob/master/src/Nulls.jl#L113) is dead simple, so I’m not really sure what can be done about this. I suppose this has something to do with allocating the iterator, though I never would have expected it to be quite this slow.

What’s our solution here? What should we be telling people to do in these cases? Is there really no substitute for explicit `null` handling?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 9, 2017, 5:05pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351/2 "2017-10-09T17:05:33Z")

</div>

Thanks for reporting this. I think that’s a case where the optimizations for `Union{T, Null}` have not been fully implemented in Julia 0.7 yet. Specifically, inlining of `next` does not happen (see [this explanation](https://discourse.julialang.org/t/small-union-types-present-and-future-looking-for-some-guidelines/6320/2) by @quinnj), which kills performance for such a simpler iterator object.

However, Julia 0.7 is about twice faster than Julia 0.6 in my tests, so that’s already a good sign (of course that’s not acceptable). Until optimizations have landed, better continue to use DataArrays, which are going to be ported to Nulls soon so that the ecosystem becomes more consistent.

We should definitely add a few test cases like this to Nulls using PkgBenchmark (or even in Base, as this is a common pattern).

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 9, 2017, 5:12pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351/3 "2017-10-09T17:12:55Z")

</div>

Thanks. If anyone around is using 0.7 and can run my script on it I would be very interested to see the results. Frankly, I am extremely eager to switch everything to Nulls because I just can’t handle the inconsistency any more, so I may have jumped the gun a bit. Fortunately, the use of union types in arrays seems surprisingly fast even in 0.6 as long as one uses methods specifically written to handle them.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 9, 2017, 6:49pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351/4 "2017-10-09T18:49:57Z")

</div>

> [@ExpandingMan](#):
>
> If anyone around is using 0.7 and can run my script on it I would be very interested to see the results.

As I said above, it’s about twice faster than 0.6 currently.

EDIT: and the version which calls `isnull` is 100× faster, and does almost no allocations.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 15, 2017, 12:40pm UTC](https://discourse.julialang.org/t/nulls-skip-is-very-slow/6351/5 "2017-10-15T12:40:46Z")

</div>

I have found a way to work around this problem: [https://github.com/JuliaData/Nulls.jl/pull/50](https://github.com/JuliaData/Nulls.jl/pull/50)

With that PR, I get:

```julia
julia> @btime for a ∈ $A
           !isnull(a) && ($x += a)
       end
  1.382 s (39601951 allocations: 604.28 MiB)

julia> @btime for a ∈ Nulls.skip($A)
           $x += a
       end
  66.971 ms (0 allocations: 0 bytes)

```