# Why can't we merge Missing and Nothing?

**URL:** <https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793>\
**Category:** Internals & Design\
**Created:** [June 28, 2019, 10:03am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793 "2019-06-28T10:03:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [June 28, 2019, 10:04am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/1 "2019-06-28T10:04:00Z")

</div>

I get that Missing and Nothing serves different purposes. They both encode that there is no particular value available, but the former indicates that usage is permitted as long as it is consistent with the multivalued state space (e.g, `true || missing = true`) while the latter indicates that usage is not permitted (most operations on `nothing` fail). A slightly contrived example of these two usecases are found within mathematics:

- no particular real value can be assign to sqrt(-1) because none of them solve x^2 = -1 (so one may return `nothing` or fail if restricted to reals).
- no particular real value can be assign to 0/0 because they all solve 0\*x = 0 (so one may return `missing` or fail).

One may observe that a single float-point value, `NaN`, covers both cases in the IEEE standard. I love the fact that Missing and Nothing are independent from other types (unlike `NaN`), but why do we need two of them? Can you find examples where merging the two types into one would cause any significant difficulty or confusion?

**EDIT**  
Rough summary can be found at [post #20](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/20).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [June 28, 2019, 10:55am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/2 "2019-06-28T10:55:26Z")

</div>

With `missing` you opt into three valued logic (like you gave as an example). I never want that because it would hide bugs in my code, so I use `nothing`.

The whole `Missing` story could kind of be a package (it used to be) with the exception that it is used in some cases like `find` on arrays which might have missing values and you would need to do type piracy to support that. `Nothing` on the other hand is quite a bit more fundamental and is ingrained much more deeply in the language (even being a part of the iteration protocol) and being the implicit return value.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 28, 2019, 11:02am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/3 "2019-06-28T11:02:28Z")

</div>

I do not know the age of the third participant, it is `missing`.  
`print` is called for its side-effect, it returns `nothing`.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [June 28, 2019, 11:05am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/4 "2019-06-28T11:05:19Z")

</div>

> [@HFriberg-mosek](#):
>
> Can you find examples where merging the two types into one would cause any significant difficulty or confusion?

Since you understand the fact that `missing` is different from `nothing` and why this is so (see [docs](https://docs.julialang.org/en/v1/manual/missing/#missing-1)), I assume you’re asking for a concrete example:  
I have a bunch of temperatures and the date I measured them at. I want to analyze these. But I messed up the measurement in a couple of those, and I don’t have the temperatures for those dates (say the thermometer broke). My analysis includes checking if the temperature is larger than some threshold and if it was taken after a certain date. Without `missing` you wouldn’t be able to run that error-free:

```julia
using Dates
temp = [2,5,3,7,missing,1]
dt = Date.(2000 .+ (1:length(temp)))
[(t > 3) | (d > Date(2000)) for (t,d) in zip(temp, dt)]

```

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [June 28, 2019, 11:24am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/5 "2019-06-28T11:24:52Z")

</div>

@kristoffer.carlsson came closest to a satisfying answer. In many cases you could merge them into `NotAvailable`:

- I do not know the age of the third participant, so it is `notavailable` .
- `print` is called for its side-effect, its return value is `notavailable` .

My (perhaps naive) take on it was that perhaps it should not be the type, but the way you use it, that determines whether it behaved as `Nothing` or `Missing`. Just like you have `skipmissing` to tell how `Missing` should be treated.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 28, 2019, 11:30am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/6 "2019-06-28T11:30:42Z")

</div>

There is an acronym for that, “DWIM” (do what I mean).

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [June 28, 2019, 11:37am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/7 "2019-06-28T11:37:29Z")

</div>

> [@HFriberg-mosek](#):
>
> My (perhaps naive) take on it was that perhaps it should not be the type, but the way you use it, that determines whether it behaved as `Nothing` or `Missing`. Just like you have `skipmissing` to tell how `Missing` should be treated.

Nahh, scratch that. I would rather have the caller actively choosing whether something should behave like `Nothing` or `Missing`. Transferring this choice of interpretation to the callee sounds like trouble. The type is a contract.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [June 30, 2019, 9:43am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/8 "2019-06-30T09:43:45Z")

</div>

> [@HFriberg-mosek](#):
>
> but why do we need two of them?

The simplest answer is that there were enough people like me that wanted something to behave like `missing` does, and enough people like Kristoffer that said having `nothing` behave this way would be terrible ([this is](https://github.com/JuliaLang/julia/pull/19034#issuecomment-255247516) probably the most suscint encapsulation of the difference is desired behavior).

But you want a specific example where we need both:

```julia
1 + missing # missing

m = match(r"foo", "bar") # nothing

1 + m # error

```

I would not want an unmatched regular expression to return `missing` (or `notavailable`), because the result is known. There’s no match. And in case I do something stupid later like try to add a number something to it, I want to be told I’m doing something stupid rather than letting it propagate.

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [July 3, 2019, 5:46pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/9 "2019-07-03T17:46:29Z")

</div>

Say I develop a package to generate data from some source (e.g., scanning webpages). When a data point cannot be fetched, should I mark it using `Missing` or `Nothing`? It seems the generator of data, not the user of this data, decides which operations are allowed on it?

My take on it would be that the user should either generate the data with a specific behavior in mind (parsing the desired type), or generate it using `Nothing` (the default) and then convert to `Missing` where appropriate. Can this conversion be done fast? Is this the most appropriate solution?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [July 3, 2019, 5:58pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/10 "2019-07-03T17:58:55Z")

</div>

> [@HFriberg-mosek](#):
>
> Say I develop a package to generate data from some source (e.g., scanning webpages). When a data point cannot be fetched, should I mark it using `Missing` or `Nothing` ?

I’d say that it depends on whether you generally expect to get the data point. If this a simple query with an a priori unknown result (which it sounds like), you should return `nothing`, just like `findfirst` and companions do for, say, arrays.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 3, 2019, 6:15pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/11 "2019-07-03T18:15:16Z")

</div>

> [@HFriberg-mosek](#):
>
> When a data point cannot be fetched, should I mark it using `Missing` or `Nothing` ?

That’s largely up to you. If you want to mark it for trying later, for example, `nothing`, or a custom object with timestamps, URIs, http status codes, and similar data could be a reasonable choice.

If you want to treat it the same way as a non-response in a survey, `missing` could work.

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [July 3, 2019, 6:52pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/12 "2019-07-03T18:52:03Z")

</div>

I hear that one should generally use `nothing` (or other appropriate types) to mark failure in data capture. Even non-response in survey could be given this type initially, as one would then be able to distinguish cases for which we are still hoping for a response (no data point is a failure) from cases where we have accepted that the response will never become available (`missing`). Ultimately the data capture must be rejected or finalized for long-term storage (in which case all unavailable data points must be `missing`).

Conclusively, the natural process for maturation of a newly generated dataset is from Union{T,Nothing} to Union{T,Nothing,Missing} to Union{T,Missing}. Any comments? I personally feel that this should be written down somewhere (anyone up for a blogpost?) with examples showing the cleanest way to make it happen.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [July 3, 2019, 7:14pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/13 "2019-07-03T19:14:02Z")

</div>

The classic distinction between `missing` and `nothing` is epistemological vs. ontological missingness. That is, `missing` represents something that _has_ a value (but you don’t know it), whereas `nothing` represents something that simply has _no_ value.

There’s some more details here, although it’s mixed with implementation design:

> **[First-Class Statistical Missing Values Support in Julia 0.7](https://julialang.org/blog/2018/06/missing/)**
>
> First-Class Statistical Missing Values Support in Julia 0.7 | The 0.7 release of Julia will soon introduce first-class support for statistical...

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [July 3, 2019, 8:12pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/14 "2019-07-03T20:12:50Z")

</div>

The way `nothing` is used in man Julia protocols illustrates this: if you call `findfirst(x, a)` and `x` does not appear in `a` at all then what index can you return? There is no correct index, so `findfirst` returns `nothing`. If on the other hand, we knew that `x` occurs somewhere in `a` but we don’t know where, then you would want to represent that with `missing`.

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [July 4, 2019, 5:15am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/15 "2019-07-04T05:15:28Z")

</div>

Let me illustrate what bothers me about having to distinguish “has a value, but it is unknown” and “has no value” in programming:

1. You may not know. If there is no information about siblings, you do not know whether to assign `NameOfSister` to `missing` (“has a value, but it is unknown”) or a `nothing` (“has no value”).

2. No standard for optional arguments because the types are loaded with meaning:  
`# Input, if you have a coupon`  
`buy(..., coupon::Union{String,Nothing}=nothing)`  
`# Input, if you know the DNA of your mom.`  
`healthanalysis(..., mom::Union{DNA,Missing}=missing) `

3. Production of `missing` types into codes that may not be programmed to handle it (kudos to @kristoffer.carlsson). Would you dare to define the generally intractable `isconvex(function)` or the generally undecidable `ishalting(callable)` as a `Union{Bool,Missing}` even though they are definitely of the “has a value, but it might be unknown” type? I never want that because the three-way-logic would hide bugs in my code.

What is more appropriate in my opinion is to distinguish _“value not available (if it exists, it is unknown)”_ and _“definitely has a value, but it is unknown”_. The former would be the standard goto type for all items in the enumeration above, whereas the second would be a specialization you opt into when needed. In practice, this is how I see `Nothing` and `Missing` already being used today.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 4, 2019, 7:12am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/16 "2019-07-04T07:12:06Z")

</div>

> [@HFriberg-mosek](#):
>
> Let me illustrate what bothers me about having to distinguish “has a value, but it is unknown” and “has no value” in programming

You should not think of `nothing` and `missing` as solutions that cover all possible scenarios. They are intended to work in some very common ones, but you are free to design your own more elaborate extensions or alternatives — what’s nice about Julia is that they will be given equal treatment (compiler optimizations, etc).

API design and data encoding are both hard problems. You should think of Julia not as a solution, but as a toolbox to iteratively build a solution.

As for your particular questions: it is hard to say more without context, but

1. I would not define `NameOfSister` at all. Perhaps a `siblings` accessor, the elements of which I could then query with `name` and `gender`, which could return `missing`.

2. For optional arguments, I would use `nothing` if that would take me to a different branch (explicitly, in the code), `missing` if I wanted to rely on generics. But that’s just my own style and I don’t follow it consistently.

3. I would not define `isconvex` and `ishalting` at all, because I don’t know how to implement them (for generic objects). In fact, I would not define function return types at all, that’s what we have a compiler for.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [July 4, 2019, 11:01am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/17 "2019-07-04T11:01:52Z")

</div>

> [@HFriberg-mosek](#):
>
> When a data point cannot be fetched, should I mark it using `Missing` or `Nothing` ?

It is pretty customary to represent an optional never-nothing field by `Union{Nothing, Typ}`, while representing partial data (e.g. unfetched data) as `Union{Missing, Typ}`. For optional fields that can be nothing, a common idiom is `Maybe{Typ}`. The third customary way of representing non-present data is by sentinel values. The built-in sentinel value for gc-managed types is nullpointer.

However, nullpointer is not officially encouraged for this purpose and pretty annoying, because it only works for gc-controlled types (not for bitstypes), which leaks abstractions and can break on upgrades, and you need to check `isassigned` before access, and you need to manipulate pointers to “unassign” a field or array entry. Nevertheless, nullpointer is the most performant way and is emitted by inner constructors that don’t set all fields, as well as `Array{T,N}(undef, sz)`. An alternative to nullpointers is to use an explicit sentinel value like `const _nil = Typ(args...)` and then check `if foo.bar === _nil` (you want triple equality here).

Given these customs, the answer would be `missing` or a more specific type like `struct FetchError code::Int32 end` that encodes information about the error (e.g. network error, parsing error, successfully fetched and parsed but pruned at later times).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 4, 2019, 11:14am UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/18 "2019-07-04T11:14:32Z")

</div>

If in doubt, use `nothing`, is would I would suggest. Even if you theoretically could have `NameOfSister` as `missing` it would still likely hide bugs by allowing you to run `1+NameOfSister` and `false && NameOfSister` without getting an error.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [July 4, 2019, 12:33pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/19 "2019-07-04T12:33:46Z")

</div>

> [@HFriberg-mosek](#):
>
> Let me illustrate what bothers me about having to distinguish “has a value, but it is unknown” and “has no value” in programming:

I’m unclear in any of your cases what’s gained by having only one option.

> [@HFriberg-mosek](#):
>
> You may not know.

This is true - is it important to distinguish between them? If so, you need 2 types, if not, pick one and document it.

> [@HFriberg-mosek](#):
>
> No standard for optional arguments because the types are loaded with meaning

This is surely better than types that have no meaning at all. If nothing (heh) is meant by `nothing`, why would it be any more sensible to use as a default argument? Again, if that’s your only option, you’re no better off using it than you are under the current state of things, and you foreclose the possibility of a type that propogates for those of us that need it.

> [@HFriberg-mosek](#):
>
> Production of `missing` types into codes that may not be programmed to handle it (kudos to @kristoffer.carlsson).

This seems like a bug, and you should raise an issue in the offending package. Or, if `missing` should not be passed to your method, don’t define methods that allow for `missing`. Users that accidentally generated `missing`s upstream will be hit with `MethodError`s (this happens to me **all. the. time** ), and then will need to make decisions about how to handle them.

Again, I don’t see how this problem is solved by only having `nothing`. In fact, it seems like I am (or people like me are) more likely to monkey patch something to handle `nothing` in a way that propogates and then silently break a bunch of code that expects `nothing` to have a specific meaning.

---

<div class="post-metadata">

**Author:** ![HFriberg-mosek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hfriberg-mosek/32/8376_2.png) [@HFriberg-mosek](https://discourse.julialang.org/u/HFriberg-mosek)\
**Post date:** [July 4, 2019, 2:02pm UTC](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/20 "2019-07-04T14:02:19Z")

</div>

Since this thread is now a wall of text and many things were not said clearly enough, I made a monologue to summarize our discussion so far:

[Me in post #1](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793): I get that `Nothing` and `Missing` serve different purposes, but can we somehow fulfill these difference usecases with just one type?  
[You in post #2](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/2): No.  
[Me in post #7](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/7): Ahh, I get it. The type is a behavioral contract. I accept the premise of two distinct types.  
[Me in post #9](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/9): But can we then distill concrete guidelines for when exactly one should use one or the other type for a specific purpose?  
[Me in post #15](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793/15): The classic interpretation to use `Missing` when _has a value (but it is unknown)_ and `Nothing` when _has no value_ is definitely flawed for the listed reasons.

Many good inputs to the discussion so far…

[Next page](https://discourse.julialang.org/t/why-cant-we-merge-missing-and-nothing/25793.md?page=2)