# Assignment of a \`missing\` value fails in DataFrames 0.11.1
**URL:** https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379
**Category:** Data
**Created:** [November 29, 2017, 4:42am UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379 "2017-11-29T04:42:09Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@mwsohn](https://discourse.julialang.org/u/mwsohn)
#### Post date: [November 29, 2017, 4:42am UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/1 "2017-11-29T04:42:09Z")
The following simple assignment fails. I think this defeats the very purpose of using DataFrames, i.e., allowing `missing` as legit values. Shouldn’t we have Array{Union{Missing,T},1} as the default type to avoid this issue?
```julia
julia> using DataFrames
julia> df = DataFrames(A = 1:4, B = ["M","F","F","M"])
julia> typeof(df[:A])
Array{Int64,1}
julia> df[3,:A] = missing
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Int64...
```
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [November 29, 2017, 4:49am UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/2 "2017-11-29T04:49:37Z")
That’s because your column doesn’t support `missing` values. You need
```julia
julia> df = DataFrames(A = Union{Int, Missing}[1, 2, 3, 4], B = ["M","F","F","M"])
```
---
### Author:  [@juliohm](https://discourse.julialang.org/u/juliohm)
#### Post date: [November 29, 2017, 5:13am UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/3 "2017-11-29T05:13:23Z")
@quinnj is there any more elegant way to initialize dataframes? Something without this explicit Union type in front maybe?
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 8:46am UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/4 "2017-11-29T08:46:23Z")
Not at this point. In the future there is the possibility that `Int?[1, 2, 3, 4]` will work, but that’s not certain yet as the `T?` syntax can also be useful for `Union{T, Void}`.
(But note that for real use cases, it’s uncommon to create data frames like this with only four rows AFAIK.)
Also, for more details about this, see the [announcement](https://discourse.julialang.org/t/dataframes-0-11-released/7296/).
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [November 29, 2017, 2:34pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/5 "2017-11-29T14:34:25Z")
Could you clarify the reason for not having `Union{T, MIssing}` be the default? Is it performance issues? If I were to, for example, read in a .csv where one column had no missing values, would DataFrames throw an error if I were to replace one of them with missing?
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 3:07pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/6 "2017-11-29T15:07:44Z")
Performance, but more importantly correctness. If your data is not supposed to contain missing values, it’s safer to use an array which does not allow for them, so that you can get an error early if for some reason code attempts to set a missing value.
Yes, when importing from CSV, columns with no missing values will buy default not allow setting missing values. It remains to be seen whether in practice it could be a problem, but I doubt it. It’s easy to replace a column instead of recoding it in place when needed, and that allows catching bugs which would introduce new missing values in columns where all values were set before.
---
### Author:  [@ssfrr](https://discourse.julialang.org/u/ssfrr)
#### Post date: [November 29, 2017, 3:17pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/7 "2017-11-29T15:17:02Z")
Another problem with automatically using `Missing` is that with the syntax in the OP, `A` and `B` are constructed before they are handed to the `DataFrame` constructor, so if `DataFrame` wanted to change the element type it would have to copy all the data into a new array. I guess if `DataFrame` already copies its data then it might be feasible to widen to `Union{T, Missing}` in the process, but that seems like a lot for default behavior to do.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [November 29, 2017, 3:35pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/8 "2017-11-29T15:35:12Z")
> [@nalimilan](#):
>
> It remains to be seen whether in practice it could be a problem, but I doubt it. It’s easy to replace a column instead of recoding it in place when needed,
A surprising number of survey softwares encode “refusals” “not applicable” etc. responses as negative numbers. I would have to create a new column just to use the command `replace x = Missing if x < 0`?
I need to download 0.7 to explore this further.
Edit: Here I am comparing DataFrames to Stata. For what it’s worth, Stata get’s around the whole problem of missing by encoding missing numbers as +Inf. Literally any solution would be better than that, because any command related to `x > Real` will apply to Missing values as well.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [November 29, 2017, 3:41pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/9 "2017-11-29T15:41:18Z")
I like the current behavior, and this seems like a non-issue to me, for example
```julia
df[:col] = convert(Vector{Union{eltype(df[:col]),Missing}}, df[:col])
```
Seems pretty simple. In most SQL databases you need to specify whether nulls are aloud, so it’s hardly unprecedented.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 3:42pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/10 "2017-11-29T15:42:51Z")
> [@pdeffebach](#):
>
> I would have to create a new column just to use the command replace x = Missing if x \< 0?
Well, in R you always create new columns, so it’s not completely unreasonable. You’d just do something like this:
```julia
df[:x] = ifelse.(df[:x] .< 0, missing, df[:x])
```
or if you want to hardcode the list of missing levels for safety:
```julia
using CategoricalArrays
df[:x] = recode(df[:x], -1 => missing, -2 => missing)
```
Of course, higher-level frameworks like Query or DataFramesMeta will allow you to do this with a nicer syntax.
EDIT: The `recode` solution currently does not work, as the returned array is allowed to contain only missing values. That could be improved. See [Add special case for single missing RHS in recode() by nalimilan · Pull Request #103 · JuliaData/CategoricalArrays.jl · GitHub](https://github.com/JuliaData/CategoricalArrays.jl/pull/103).
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [November 29, 2017, 3:46pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/11 "2017-11-29T15:46:28Z")
On second thought, I can warm to this idea. It is definitely more reproducible to not alter the underlying data by introducing missings.
Is it possible to have subtypes of missing values?
`struct Refuse <: Missing` ?
Stata currently offers types that display as `.d`, `.a`, or `.r`. However if you are working with an old dataset, It’s difficult to know what those one-letter missings mean. This is a problem I am currently having with Stata.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 3:48pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/12 "2017-11-29T15:48:15Z")
For now there’s no `AbstractMissing` type, but any type can overload `ismissing`, which should be enough for most (if not all) purposes. I have considered creating a flavoured `Missing` type which would allow specifying the kind of missing values, but that’s lower in the priority list than getting the basics to work.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [November 29, 2017, 5:05pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/13 "2017-11-29T17:05:40Z")
It would be somewhat advanced but it’s not entirely implausible for the compiler to become smart enough to recognize that you’re copying a vector to another one with the same memory layout and not keeping any references to the original, which allow just reusing the memory without any copying while retaining the clearer semantics.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [November 29, 2017, 6:16pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/14 "2017-11-29T18:16:50Z")
> [@nalimilan](#):
>
> Performance, but more importantly correctness. If your data is not supposed to contain missing values, it’s safer to use an array which does not allow for them, so that you can get an error early if for some reason code attempts to set a missing value.
FWIW, I’d also be in favor of defaulting into `Union{Int, Missing}` rather than `Int`. **[EDIT: or allow on-the-fly conversions when one attempts to assign missing values.]** I recognize there may be performance implications, but I’m not sure about the correctness argument. I guess my sense was that the aim of DataFrames was, in part, to accommodate real world data, which often has the possibility of being missing, even if there were not missing values in original read-in. Seems reasonable that by default one allows missings.
I also do frequently set things to missing in columns that don’t originally have missing data. If, for example, I realize there’s a subpopulation for which I don’t think my survey data on a given response is valid (say I realized my enumerator didn’t understand the question he was supposed to ask, or I have some GPS coordinates that turn out to have been hand-entered wrong and are in the wrong country), I want to change them to missing.
And I also frequently have data that hard-codes missings as sentinel values like -1 that I later want to swap for missings.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [November 29, 2017, 6:50pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/15 "2017-11-29T18:50:23Z")
> [@nickeubank](#):
>
> I guess my sense was that the aim of DataFrames was, in part, to accommodate real world data, which often has the possibility of being missing, even if there were not missing values in original read-in. Seems reasonable that by default one allows missings.
It depends on what data you are dealing with. In a lot of cases data is stored using `NaN` and empty strings anyway, in which case you’d have to set up some parsing to convert them to `missing` anyway. In other cases the data is effectively read-only. The point is, you never really know, so shouldn’t the default behavior be _not_ to change the type of a column rather than to change it?
Also, for what it’s worth: I’ve been using DataFrames.jl for quite a while through several manifestations, and I find that I _strongly_ prefer there to be no special handling of the columns. It’s just so much cleaner that way. And since the columns are just `AbstractVectors`, and not some special data structure (to me this is basically the best feature of DataFrames.jl), why not just decide what you want when you insert the column in the first place? If you guess wrong, it’s no big deal, you can just convert it.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 6:52pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/16 "2017-11-29T18:52:05Z")
I should have noted that you can call `allowmissing!(df, col)` to make a column accept missing values. I have just filed a [pull request](https://github.com/JuliaData/DataFrames.jl/pull/1292) so that you can also do `allowmissing!(df)`, which will apply this change to all columns.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [November 29, 2017, 7:11pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/17 "2017-11-29T19:11:29Z")
> [@nalimilan](#):
>
> I should have noted that you can call allowmissing!(df, col) to make a column accept missing values. I have just filed a pull request so that you can also do allowmissing!(df), which will apply this change to all columns.
That’s great, thanks for pointing that out!
Here’s a related question: how hard would it be to create a promotion rule for DataFrame columns that allowed, say, an `Int` to be promoted to a `Union{Int, Missing}` when someone tries to assign a `missing` to an entry? That seems like it meet’s @ExplodingMan’s desire to avoid different default typing, but also addresses @mwsohn (and my) desire to make using `missing` seamless?
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [November 29, 2017, 7:32pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/18 "2017-11-29T19:32:41Z")
> [@nickeubank](#):
>
> Here’s a related question: how hard would it be to create a promotion rule for DataFrame columns that allowed, say, an Int to be promoted to a Union{Int, Missing} when someone tries to assign a missing to an entry? That seems like it meet’s @ExplodingMan’s desire to avoid different default typing, but also addresses @mwsohn (and my) desire to make using missing seamless?
That’s not really possible. When you do `df[:x][1] = missing`, you are calling `setindex!` on the vector, so `DataFrame` is not involved at all, and Julia (rightly) does not allow changing the type of a binding from inside a function which takes it as an argument. OTOH `df[:x, 1] = missing` could replace the column with a new vector accepting missing values, but that would be very surprising.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [November 29, 2017, 7:40pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/19 "2017-11-29T19:40:06Z")
Could you clarify what you mean by surprising?
From the user’s perspective, as long as missing values are propagated in a reasonable way, this wouldn’t be unexpected.
---
### Author:  [@Nosferican](https://discourse.julialang.org/u/Nosferican)
#### Post date: [November 29, 2017, 8:18pm UTC](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379/20 "2017-11-29T20:18:33Z")
I usually make all columns allow missing or pure one Type depending on the operation. For example, for first-difference which I know will lead to missing values I allow missing for all columns. For any transformation that drops the missing values I keep them one Type.
[Next page](https://discourse.julialang.org/t/assignment-of-a-missing-value-fails-in-dataframes-0-11-1/7379.md?page=2)