# Replacing missing, really

**URL:** <https://discourse.julialang.org/t/replacing-missing-really/7952>\
**Category:** Data\
**Created:** [December 23, 2017, 11:23pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952 "2017-12-23T23:23:59Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![harven](https://avatars.discourse-cdn.com/v4/letter/h/3da27b/32.png) [@harven](https://discourse.julialang.org/u/harven)\
**Post date:** [December 23, 2017, 11:23pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/1 "2017-12-23T23:23:59Z")

</div>

How can I replace the missing value in a dataframe by, say, 0?

There is a `Missings.replace` function but it does not actually make the replacement. It builds a lazy data structure `Missings.EachReplaceMissing{DataFrames.DataFrame,Int64}...` that most functions working with dataframes (e.g. plots) can’t handle at the moment. Collecting such a structure does not produce a dataframe but gives a MethodError.

---

<div class="post-metadata">

**Author:** ![harven](https://avatars.discourse-cdn.com/v4/letter/h/3da27b/32.png) [@harven](https://discourse.julialang.org/u/harven)\
**Post date:** [December 24, 2017, 1:08pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/2 "2017-12-24T13:08:22Z")

</div>

To answer my own question, I went with

```
  for col in names(df)
    df[ismissing.(df[col]), col] = 0
  end

```

but I guess that there may be a simpler way.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 24, 2017, 1:30pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/3 "2017-12-24T13:30:17Z")

</div>

`Missings.replace` expects an iterable, but a `DataFrame` isn’t one.

I would do something like you did above. Note that `DataFrame`s may not have a homogeneous column type, so replacing missing values in all columns with the same value may not be a common operation.

Alternatively,

```julia
using DataFrames
import Missings
df = DataFrame(a = [1, missing, 2], b = [missing, 3, 4])
df2 = map(c -> collect(Missings.replace(c, 0)), eachcol(df))

```

also works.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [December 24, 2017, 2:19pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/4 "2017-12-24T14:19:39Z")

</div>

@Tamas_Papp’s solution has the advantage that the returned `DataFrame` contains columns which do not allow for missing values, which will be faster. On the contrary, only replacing missing values with `0` won’t change the type of columns, though you could use `disallowmissing` to do that manually.

An alternative approach is to use `coalesce` (which is included in Base in Julia 0.7, but only in Missings in Julia 0.6):

```julia
for col in names(df)
   df[col] = Missings.coalesce.(df[col], 0)
end

```

Do you know of simpler ways of doing this in other software?

---

<div class="post-metadata">

**Author:** ![harven](https://avatars.discourse-cdn.com/v4/letter/h/3da27b/32.png) [@harven](https://discourse.julialang.org/u/harven)\
**Post date:** [December 24, 2017, 5:36pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/5 "2017-12-24T17:36:36Z")

</div>

Thanks for the answer. As far as I remember, this is done in R with

```
df[is.na(d)] <- 0

```

And now I realize that a similar syntax can be used in julia with the help of the `eachcol` iterator.

```
  for (_, col) in eachcol(df)
    col[ismissing.(col)] = val
  end

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 24, 2017, 5:57pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/6 "2017-12-24T17:57:21Z")

</div>

In Stata you can use a function called `mvencode`

```julia
mvencode variable, mv(0)

```

But I hate it when people use it. If I’m reading a script I would rather see

```julia
replace variable = 0 if missing(variable)`

```

because it mirrors other replacements.

Maybe there are performance benefits to using `mvencode` but I’ve never noticed anything. With that in mind, @harven’s answer above is nice because it keeps the same syntax as other replacements.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [December 25, 2017, 10:46pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/7 "2017-12-25T22:46:54Z")

</div>

OK. The difference we have compared to R is that we support arrays which do not accept missing values, so there are more possible solutions than in R depending on the use case. Also I don’t think we want to allow `df[ismissing.(df)] = 0`, because data frames are not matrices. Anyway it’s a terribly inefficient approach since it forces you to allocate a matrix of the size of the dataframe.

`mvencode` appears to work variable by variable IIUC, so that’s more or less similar to what we have.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 17, 2019, 2:10am UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/8 "2019-01-17T02:10:58Z")

</div>

Can’t we just do this?

replace(df, missing=\>0)

---

<div class="post-metadata">

**Author:** ![kmundnic](https://avatars.discourse-cdn.com/v4/letter/k/e19adc/32.png) [@kmundnic](https://discourse.julialang.org/u/kmundnic)\
**Post date:** [February 26, 2020, 11:20pm UTC](https://discourse.julialang.org/t/replacing-missing-really/7952/9 "2020-02-26T23:20:15Z")

</div>

Note that it’s now possible to do:

```julia
coalesce.(df, 0)

```