# Missing data and NamedTuple compatibility

**URL:** <https://discourse.julialang.org/t/missing-data-and-namedtuple-compatibility/8136>\
**Category:** Internals & Design\
**Created:** [January 3, 2018, 5:52pm UTC](https://discourse.julialang.org/t/missing-data-and-namedtuple-compatibility/8136 "2018-01-03T17:52:39Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [January 4, 2018, 3:57am UTC](https://discourse.julialang.org/t/missing-data-and-namedtuple-compatibility/8136/5 "2018-01-04T03:57:41Z")

</div>

> [@davidanthoff](#):
>
> If `{}` in Query.jl mapped to NullableNamedTuple (or would it be MissableNamedTuple 😉 ) you would always get a result where every column supports missing values, even if you started with a source that didn’t have any support for missing values.

My idea was to take care of that when collecting. When the iterable of NullableNamedTuples (a better name is needed…) is collected into say a DataFrame, the columns that do not have any `missings` are converted to regular `Vector{T}` whereas the others stay as `Vector{Union{T, Missing}}`. This conversion could maybe be possible without copying, though I’m not sure about that.

Inside the query, it is true that all columns would accept Missings, but that shouldn’t be a concern: a key advantage of the Missing approach (over a container approach) is that the code doesn’t need to be changed if some column allows missing data if there actually is no missing data (whereas `DataValue` would require the occasional `get` as soon as it encounters a function that is not “whitelisted”).

One might argue that this solution is still not ideal as in the final output some columns that “should be nullable” in the sense that they are a function of nullable columns would not be nullable if there is no missing data in the input. I’m not sure whether this is a problem in practice (though I don’t think it should be). I also don’t think there is a way to avoid this behavior with a `Union` approach to missing data without relying on `Base._return_type`.

> [@davidanthoff](#):
>
> I think the reality of all of this is that we’ll simply have a data stack that uses two different missing data implementations (Query.jl and friends will just continue to use DataValue)

What drove my curiosity was this [comment](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/44) about JuliaDB. I was trying to understand what solution JuliaDB’s developers had in mind as it seems to me that the same issues that affect Query would also affect JuliaDB (JuliaDB’s `map` being very similar to Query’s `@map`).

---

_[View the full topic](https://discourse.julialang.org/t/missing-data-and-namedtuple-compatibility/8136)._