# Why is missing defined as singleton

**URL:** <https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561>\
**Category:** General Usage\
**Created:** [December 21, 2019, 8:42pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561 "2019-12-21T20:42:38Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jandehaan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jandehaan/32/6805_2.png) [@jandehaan](https://discourse.julialang.org/u/jandehaan)\
**Post date:** [December 21, 2019, 8:42pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/1 "2019-12-21T20:42:38Z")

</div>

Currently type Missing is defined as `struct Missing end`  
and the singleton instance of that type is defined as  
`const missing = Missing()`

But what if we defined the following:

```julia
struct Missing
   why::Int
end
Missing() = Missing(1)
const missing = Missing()

```

`missing` would no longer be a singleton, but now we could capture  
the reason why a value is missing. For example

```julia
const NI = Missing(1) # no information
const INV = Missing(11) # invalid
const DER = Missing(111) # derived
const OTH = Missing(112) # other
const NINF = Missing(1121) # negative infinite
const PINF = Missing(1122) # positive infinite
const UNC = Missing(113) # uncoded
const MSK = Missing(12) # masked
const NA = Missing(13) # not applicable
const UNK = Missing(14) # unknown
const ASKU = Missing(141) # asked but unknown
const NASK = Missing(142) # not asked
const NAVU = Missing(143) # not available
const NAV = Missing(1431) # temporarily unavailable
const QS = Missing(144) # sufficient quantity
const TRC = Missing(145) # trace
const NP = Missing(2) # not present

myvalue = ASKU
if ismissing(myvalue) ...

```

Back to the original question: why has `missing` been defined as a singleton?

---

<div class="post-metadata">

**Author:** ![longemen3000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/longemen3000/32/7298_2.png) [@longemen3000](https://discourse.julialang.org/u/longemen3000)\
**Post date:** [December 21, 2019, 8:55pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/2 "2019-12-21T20:55:06Z")

</div>

> [@jandehaan](#):
>

for what i read [here](https://julialang.org/blog/2018/06/missing), when you have an `x = Vector{Union{Missing,Float64}}` a second array is created, of the same size, holding information of when the value at position x[i] is missing or not. using the approach would imply storing that information somewhere, with the subsequent cost.  
The answer to your question is mencioned explicitly:

> One of Julia’s strengths is that user-defined types are as powerful and fast as built-in types. To fully take advantage of this, missing values had to support not only standard types like `Int` , `Float64` and `String` , but also any custom type. For this reason, Julia cannot use the so-called _sentinel_ approach like R and Pandas to represent missingness, that is reserving special values within a type’s domain. For example, R represents missing values in integer and boolean vectors using the smallest representable 32-bit integer ( `-2,147,483,648` ), and missing values in floating point vectors using a specific `NaN` payload ( `1954` , which rumour says refers to Ross Ihaka’s year of birth). Pandas only supports missing values in floating point vectors, and conflates them with `NaN` values.  
> In order to provide a consistent representation of missing values which can be combined with any type, Julia 0.7 will use `missing` , an object with no fields which is the only instance of the `Missing` singleton type. This is a normal Julia type for which a series of useful methods are implemented. Values which can be either of type `T` or missing can simply be declared as `Union{Missing,T}` . For example, a vector holding either integers or missing values is of type `Array{Union{Missing,Int},1}` :

---

<div class="post-metadata">

**Author:** ![jandehaan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jandehaan/32/6805_2.png) [@jandehaan](https://discourse.julialang.org/u/jandehaan)\
**Post date:** [December 22, 2019, 2:39am UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/3 "2019-12-22T02:39:50Z")

</div>

@longemen3000 thank you for the link to the blog article. It seems an opportunity was lost. Where the 2nd array indicates that the value is missing, the corresponding value in the 1st array is uninitialized. That is a pity. That slot in the 1st array could have been used to indicate why the data is missing. Maybe this design could be revisited at some point in the future.

---

<div class="post-metadata">

**Author:** ![longemen3000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/longemen3000/32/7298_2.png) [@longemen3000](https://discourse.julialang.org/u/longemen3000)\
**Post date:** [December 22, 2019, 3:42am UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/4 "2019-12-22T03:42:40Z")

</div>

There are some problems with storing a payload in my opinion with that method:

1. missings in any type implies the use of missings on struts with variable length of bits (as Julia 1.3, there is a minimum length of 8 bits to make a type, but that can change in the future.

2. The missing propagation proposed here sounds a lot like NaN propagation. NaNs are generated on numerical errors,like `log(-3.0)`. Here it makes sense to create a Nan with a payload representing the error and propagate those. As far as I know, Julia does not have NaN propagation, and I don’t know about any existing implementation on popular libraries of NaN propagation apart from sentinel values. Furthermore, NaN propagation can only be used with Floats.

3. For what I understand, the missing concept is different from the NaN concept. The first is a notion of statistical missingness, whereas the presence of a NaN definitely represents a numerical error somewhere on your numerical program.

With that said, still an alternate implementation can be done. Where the payload is in the remaining 7 bits of the hidden array (using 1 but to represent true or false)

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 22, 2019, 3:48am UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/5 "2019-12-22T03:48:18Z")

</div>

You could probably use this package to represent reason for missingness?  
[https://github.com/meggart/SentinelMissings.jl](https://github.com/meggart/SentinelMissings.jl)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 22, 2019, 7:40am UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/6 "2019-12-22T07:40:13Z")

</div>

> [@jandehaan](#):
>
> It seems an opportunity was lost.

Not really — you can just define your own type for missingness with a payload, implement the relevant methods, and it will be equally efficient as an implementation in `Base` would be. You can of course take advantage of existing functions (eg `ismissing`, etc) by defining methods for them. That’s a great thing about Julia: there is no “secret sauce”, user code is on equal footing with “built-in” constructs.

That said, I think the main reason to be cautious about missing values with a payload is that you have to define semantics for all operations, eg `Missing(1) + Missing(2)`. I don’t think there is a canonical way of doing this, so I think that not doing this in `Base` was the right choice: the details should be worked out in a package.

---

<div class="post-metadata">

**Author:** ![jandehaan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jandehaan/32/6805_2.png) [@jandehaan](https://discourse.julialang.org/u/jandehaan)\
**Post date:** [December 22, 2019, 3:00pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/7 "2019-12-22T15:00:46Z")

</div>

> [@Tamas\_Papp](#):
>
> define semantics for all operations, eg `Missing(1) + Missing(2)`

Yes, there is no canonical way to propagate missing values, but I believe that the current implementation would return `missing` in all cases. In my example above that would equate to the default `Missing(1) # No Information`, which seems like a reasonable approach to me. But I can see why someone would like to be more sophisticated with respect to the propagation of missing values.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 22, 2019, 3:22pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/8 "2019-12-22T15:22:13Z")

</div>

Your idea is interesting! It’s been discussed before. Having missing values that represent “do’t know”, “not asked” etc. would be really useful, particularly for the analysis of household surveys. I understand this is also really important with medical data.

Stata has this functionality with a `.r` and `.d` missing types, which are encoded as sentinel values.

I think this would be a good feature to have, but in the implementation it would be tough to keep performance.

Additionally, your implementation with sentinal values is a bit odd. If it were to be implemented it would probably be via an abstract type

```julia
abstract type AbstractMissing end
struct Missing <: AbstractMissing end
struct DontKnowMIssing <: AbstractMissing end

```

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [December 26, 2019, 9:38pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/9 "2019-12-26T21:38:47Z")

</div>

Actually I had in mind the extension to typed missing values when we implemented `missing`. It’s just not been a priority, since it sounds essential to get the basic support right first.

There are two ways to implement “typed” or “flavoured” missing values in a fully generic way (i.e. without sentinels): with one type for each kind of missing value as @pdeffebach suggested, or as a single `TypedMissing` type holding an `Int` field indicating the kind (as an enum) as @longemen3000 suggested. The advantage of the former is that the compiler can optimize the storage and computation of such values, so that they would be as efficient as `Missing`. The drawback is that it’s going to recompile functions for each kind of missing value, which is costly if you use several kinds.

That or those type(s) could be defined in any package (Missings.jl would be a good home). Though currently there’s a limitation as Julia Base uses `x === missing` instead of `ismissing(x)` in some places for performance reasons (see [this issue](https://github.com/JuliaLang/julia/issues/27681)), so Julia will have to be adapted a bit for all operations to work (notably `==` on arrays).

FWIW, there’s at least one standard providing a list of possible kinds/reasons for missingness: [NullFlavor - FHIR v4.0.1](http://www.hl7.org/fhir/v3/NullFlavor/cs.html)

---

<div class="post-metadata">

**Author:** ![croberts](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/croberts/32/9465_2.png) [@croberts](https://discourse.julialang.org/u/croberts)\
**Post date:** [August 25, 2022, 3:57pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/10 "2022-08-25T15:57:01Z")

</div>

I’d like to bump this 3-year old comment for visibility.

There have been multiple instances where I’ve wanted functions to dispatch on special cases in missing. Variables that are missing due to being out-of-sample (such as for unbalanced panels) are one such case.

If

```julia
abstract type AbstractMissing end
struct Missing <: AbstractMissing end

```

And all functions currently defined on ::Missing were instead defined on ::AbstractMissing, then it would have made other code I’ve written much simpler. As things are, I need to either do type piracy (a big no no) or I need to create a new type and manually extend many other functions that currently handle missings.

Can we put this on the todo list for future versions of Julia? It should be a non-breaking change given how multiple dispatch works.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [August 25, 2022, 6:20pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/11 "2022-08-25T18:20:57Z")

</div>

> [@croberts](#):
>
> I need to create a new type and manually extend many other functions that currently handle missings

This seems like the correct solution. Why shouldn’t you do this (other than you wish someone else had already done it)?

---

<div class="post-metadata">

**Author:** ![croberts](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/croberts/32/9465_2.png) [@croberts](https://discourse.julialang.org/u/croberts)\
**Post date:** [August 26, 2022, 7:51pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/12 "2022-08-26T19:51:53Z")

</div>

> [@tbeason](#):
>
> This seems like the correct solution. Why shouldn’t you do this (other than you wish someone else had already done it)?

I would be happy to make a new type:  
` struct MyMissingType <: AbstractMissing end`  
and then extend methods for only the few relevant special cases for this type.  
Using Holy Traits and defining functions on abstract types makes extension much simpler.  
If, for example, I wanted skipmissing to work with MyMissingType as it does with the default Missing, this would be easy if the helper functions for skipmissing were defined on AbstractMissing. I would have to do nothing at all.

But because there is no AbstractMissing type, I would need to reimplement all functions currently defined on Missing to also work with MyMissingType. This is time consuming and results in a large amount of boilerplate. In practice, I do not reimplement all functions, which means that MyMissingType does not support all operations for Missing types. Lazy.jl can help automate this, but in my experience it is not terribly reliable and is not as clean as simply subtyping an abstract type. These problems are acceptable if I am writing code I will not share with others, but is not so great if I intend to write a general-use package. More generally, it is not consistent with julia’s multiple dispatch paradigm (and it’s “unreasonable effectiveness”) for me to write an independent missingness type hierarchy when base Julia already supports missingness. It’s much better to subtype an extant abstract type.

The power of multiple dispatch is one of the things I love about Julia. Unfortunately, when base julia creates concreate types that are not subtypes of abstract types (or does not define generic functions in terms of the abstract type), a lot of the power of multiple dispatch is lost.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [August 26, 2022, 8:27pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/13 "2022-08-26T20:27:07Z")

</div>

@nalimilan has been working on this

> **[GitHub - nalimilan/TypedMissings.jl: Support for different kinds of missing...](https://github.com/nalimilan/TypedMissings.jl)**
>
> Support for different kinds of missing values in Julia - GitHub - nalimilan/TypedMissings.jl: Support for different kinds of missing values in Julia

---

<div class="post-metadata">

**Author:** ![croberts](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/croberts/32/9465_2.png) [@croberts](https://discourse.julialang.org/u/croberts)\
**Post date:** [August 26, 2022, 8:52pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/14 "2022-08-26T20:52:58Z")

</div>

Thanks!

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [August 26, 2022, 9:06pm UTC](https://discourse.julialang.org/t/why-is-missing-defined-as-singleton/32561/15 "2022-08-26T21:06:01Z")

</div>

Yeah, the package is mostly functional but it currently has to overwrite methods in Julia Base that hardcode `=== missing` instead of using `ismissing`. I started fixing this in Base, but got blocked by a performance regression at [Use `ismissing(x)` instead of `x === missing` by nalimilan · Pull Request #44407 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/44407) and I’m not qualified to fix that in the compiler. There are also other issues that require a deeper redesign in Base, notably `promote_typejoin` ([Allow custom types to get narrow Union type with map (like `promote_typejoin` for `Missing`/`Nothing`) · Issue #38241 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/38241)).