# Changing missing logic

**URL:** <https://discourse.julialang.org/t/changing-missing-logic/32449>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 18, 2019, 4:01pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449 "2019-12-18T16:01:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [December 18, 2019, 4:01pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/1 "2019-12-18T16:01:56Z")

</div>

I really like to use `missing`, but there are _some_ instances where it becomes annoying. First, logical indexing. Second, when a package hasn’t implemented a safety method `f(missing)=missing`.

For example, suppose I read in a DataFrame from a file and that file contains missing values. Now I want to subset using `where` or `in` on one of the columns containing missing values. The understandably conservative logic of `missing` requires two steps rather than one. First, I must eliminate rows with `missing`, then I can do the filtering operation.

```julia
using DataFrames
df = DataFrame(a=[1,missing,3],b=["low",missing,"high"])
@where(df,:a .> 0) # error
@where(df,in.(:b,Ref(("low","high")))) # error

# necessary (?)
@where(df,.!ismissing.(:a),:a.>0)

```

Is there a way to change the logic so that `missing > 0 == false` or `missing \in (0,1) == false`? I’m not proposing to change the default behavior, I’m just wondering if there is a way for me to basically make `missing` behave more like `NaN` but not just for numeric columns.

Additionally, and not to pick on any package specifically, but to give an example.

```julia
using Distributions
pdf(Normal(0,1),missing) # errors

```

What am I to do in this example? Well, this again becomes a two step procedure rather than one. I need to define an anonymous function like `pdf2(x) = ismissing(x) ? missing : pdf(Normal(0,1),x)` or something.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 18, 2019, 4:22pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/2 "2019-12-18T16:22:31Z")

</div>

For your first example, yeah I think that is would be the right way to keep all observations that aren’t missing and greater than `0`. I agree that the `.!ismissing.(:a)` is not the most convenient syntax. A convenient workaround would be to simply define

```julia
notmissing(x) = ismissing(x) == false

```

The way to deal with your second example is though the function `passmissing` defined in [Missings.jl](https://github.com/JuliaData/Missings.jl). It is a mapping `Function -> Function` that returns `missing` more frequently.

For your last example you would do

```julia
julia> passmissing(pdf)(Normal(0, 1), missing)
missing

```

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [December 18, 2019, 4:26pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/3 "2019-12-18T16:26:25Z")

</div>

> [@tbeason](#):
>
> @where(df,.!ismissing.(:a),:a.\>0)

I sometimes use `@where(df, coalesce.(:a.>0, false))`

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [December 18, 2019, 4:40pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/4 "2019-12-18T16:40:01Z")

</div>

I was not aware of `passmissing`, which does look like a nice utility. That still basically amounts to my existing solution just in a much more compact way.

The issue with the first is not the `.!ismissing.(:a)` is ugly. It is that it is necessary before trying `:a .> 0`. Not terrible if you are just doing this for one column, but if you are chaining things together it gets out of hand. My DataFramesMeta `@linq` calls basically now have one long line of `where(.!ismissing.(:a),.!ismissing.(:b), ....) |>` before I move on to actual filtering.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [December 18, 2019, 4:40pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/5 "2019-12-18T16:40:50Z")

</div>

That does work better for that situation but I can’t seem to get it to work for the second scenario using `in`

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 18, 2019, 4:42pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/6 "2019-12-18T16:42:34Z")

</div>

That’s a good point. Maybe do this?

```julia
ulia> @where(df, notmissing.(:a.>0))

```

It’s not good that the order matters here. You should file an issue with DataFramesMeta and see if there is anything that should be changed.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [December 18, 2019, 8:45pm UTC](https://discourse.julialang.org/t/changing-missing-logic/32449/7 "2019-12-18T20:45:30Z")

</div>

You could also use [Query.jl](https://github.com/queryverse/Query.jl) for this. It uses [DataValues.jl](https://github.com/queryverse/DataValues.jl) for missing value handling, which is based on the C# semantics for missing values, so your first examples just work like

```julia
df |> @filter(_.a > 0)

```

and

```julia
df |> @filter(_.b in ("low", "high"))

```

. If you want a `DataFrame` with your results, you just pipe it into one:

```julia
df |> @filter(_.a > 0) |> DataFrame

```