# Inconsistent (unexpected?) behavior in elementwise operators

**URL:** <https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560>\
**Category:** General Usage\
**Created:** [December 6, 2017, 9:52am UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560 "2017-12-06T09:52:24Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [December 6, 2017, 9:52am UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/1 "2017-12-06T09:52:24Z")

</div>

I have a `ratings` variable:

```julia
ratings = 10×3 DataFrames.DataFrame
│ Row │ Movie title │ Dean │ Sam │
├─────┼─────────────────────────────────────┼──────┼─────┤
│ 1 │ "Moonlight (2016)" │ 10 │ 0 │
│ 2 │ "Zootopia (2016)" │ 0 │ 0 │
│ 3 │ "Arrival (2016)" │ 10 │ 10 │
│ 4 │ "Hell or High Water (2016)" │ 10 │ 0 │
│ 5 │ "La La Land (2016)" │ 9 │ 0 │
│ 6 │ "The Jungle Book (2016)" │ 2 │ 0 │
│ 7 │ "Manchester by the Sea (2016)" │ 8 │ 0 │
│ 8 │ "Finding Dory (2016)" │ 4 │ 0 │
│ 9 │ "Captain America: Civil War (2016)" │ 6 │ 9 │
│ 10 │ "Moana (2016)" │ 0 │ 0 │

```

I then apply some elementwise filtering to it:

```julia
recommended_movies = ratings[Array(ratings[:Dean] .> 7) .& Array(ratings[:Sam] .== 0), :]

```

The above code, where I explicitly use `7` works as expected.

However, if I do `const minimum_rating = 7` and then

```julia
recommended_movies = ratings[Array(ratings[:Dean] .> minimum_rating) .& Array(ratings[:Sam] .== 0), :]

```

it fails with

```julia
ERROR: LoadError: MethodError: no method matching isless(::Int64, ::Nullable{Int64})
Closest candidates are:
  isless(!Matched::Nullable{Union{}}, ::Nullable) at nullable.jl:235
  isless(!Matched::DataArrays.NAtype, ::Any) at /Users/adrian/.julia/v0.6/DataArrays/src/operators.jl:383
  isless(::Real, !Matched::AbstractFloat) at operators.jl:97

```

🤐

It seems to me that this is the kind of refactoring that should Just Work™

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [December 6, 2017, 3:13pm UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/2 "2017-12-06T15:13:49Z")

</div>

It’s really subtle, but Julia does actually treat those two cases differently. What’s happening is that the built-in broadcasting mechanism actually inlines literals into the function being broadcast. So, for example, if you do:

```julia
f.(x, y)

```

that turns into `broadcast(f, x, y)`, but if you do:

```julia
f.(x, 3)

```

it’s turned into something like `broadcast(z -> f(z, 3), x)`. You can see this by comparing `expand(:(f.(x, y)))` to `expand(:(f.(x, 3)))`.

That said, I can’t actually reproduce the result you’ve shown (at least without knowing exactly how you constructed your data frame and which version of DataFrames this is). But my guess is that you’re essentially asking for two levels of broadcasting: one for the elementwise operation and one for the unwrapping of Nullables, but somehow due to the specifics of broadcasting in your version of DataFrames, the inlining of the literal 7 lets the first case work.

In general, elementwise comparison against a vector of Nullables doesn’t work:

```julia
julia> x = Nullable.(1:3)
3-element Array{Nullable{Int64},1}:
 1
 2
 3

julia> x .> 2
ERROR: MethodError: no method matching isless(::Int64, ::Nullable{Int64})
Closest candidates are:
  isless(::Nullable{Union{}}, ::Nullable) at nullable.jl:235
  isless(::Missings.Missing, ::Any) at /Users/rdeits/.julia/v0.6/Missings/src/Missings.jl:77
  isless(::Real, ::AbstractFloat) at operators.jl:97
  ...
Stacktrace:
 [1] (::##29#30)(::Nullable{Int64}) at ./<missing>:0
 [2] broadcast_t(::Function, ::Type{Any}, ::Tuple{Base.OneTo{Int64}}, ::CartesianRange{CartesianIndex{1}}, ::Array{Nullable{Int64},1}) at ./broadcast.jl:256
 [3] broadcast_c at ./broadcast.jl:319 [inlined]
 [4] broadcast(::Function, ::Array{Nullable{Int64},1}) at ./broadcast.jl:434

```

Instead, we have to add another layer of broadcasting:

```julia
julia> (z -> z .> 2).(x)
3-element Array{Nullable{Bool},1}:
 false
 false
 true

```

Fortunately, this is a situation that will be resolved by the shift from `Nullable{T}` to `Union{T, Missing}`:

```julia
julia> x = Union{Int, Missing}[1:3...]
3-element Array{Union{Int64, Missings.Missing},1}:
 1
 2
 3

julia> x .> 2
3-element BitArray{1}:
 false
 false
  true

julia> x[2] = missing
missing

julia> x .> 2
3-element Array{Any,1}:
 false
      missing
  true

```

Note that the last result is an `Array{Any}` because I’m on Julia v0.6.1 which lacks the Union optimizations of Julia master.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [December 6, 2017, 3:19pm UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/3 "2017-12-06T15:19:15Z")

</div>

For what it’s worth, I strongly suggest you upgrade to DataFrames 0.11.1. This is exactly the sort of thing that is a non-issue in the new version.

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [December 6, 2017, 3:56pm UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/4 "2017-12-06T15:56:02Z")

</div>

Thank you, that is quite some interesting insight, very instructive. I’m on DataFrames 0.10.1 still.

Just out of curiosity, Is the `Missing` type defined by `DataFrames` or by Julia itself (like `Nullable`)? Are there plans for dropping `Nullable` in favor of `Union` and `Missing` at language level?

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [December 6, 2017, 3:59pm UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/5 "2017-12-06T15:59:03Z")

</div>

Thank you, yes, I’m planning to - however, the recommendation upon the release was to install from scratch, per the release announcement:

> Also note that until all packages on your local installation have been ported to DataFrames 0.11.0, they will keep requiring version 0.10.1, and the package manager will not update DataFrames to version 0.11.0. If removing the problematic dependencies is not an option, you can use a separate Julia package directory to test the new framework: just set the JULIA\_PKGDIR before starting Julia, and run Pkg.add(“DataFrame”).

On my current setup, it wasn’t updated yet and I’m reluctant to lose dependent packages that haven’t been updated yet.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [December 6, 2017, 4:52pm UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/6 "2017-12-06T16:52:41Z")

</div>

`Missing` is defined in [Missings.jl](https://github.com/JuliaData/Missings.jl) but is re-exported by the new DataFrames, so `using DataFrames` will bring it into scope. And yes, `Nullable` is likely to be moved out of Base: [https://github.com/JuliaLang/julia/issues/22682](https://github.com/JuliaLang/julia/issues/22682) (but will continue to live on in [Nullables.jl](https://github.com/JuliaArchive/Nullables.jl)) for anyone who still needs that type.

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [December 7, 2017, 8:35am UTC](https://discourse.julialang.org/t/inconsistent-unexpected-behavior-in-elementwise-operators/7560/7 "2017-12-07T08:35:13Z")

</div>

Got it, thank you