# Replace missing with 0.0 in dataframe

**URL:** <https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 8, 2019, 8:02pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029 "2019-12-08T20:02:01Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [December 8, 2019, 8:02pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/1 "2019-12-08T20:02:01Z")

</div>

I am trying to replace missing values in Float coluns with 0.0.

My starting point:

```julia
for col in names(df)
    df[ismissing.(df[col]), col] = 0.0
end

```

While this works, it has two weak points:  
a. it doesn’t check the type of the column; only in Float colums  
missing shall be replaced with 0.0  
b. it throws a warning

Any idea how to fix these two issues?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [December 8, 2019, 8:57pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/2 "2019-12-08T20:57:51Z")

</div>

The warnings can easily be avoided:

```julia
julia> for col in names(df)
           df[ismissing.(df[:,col]), col] .= 0.0
       end

```

and now I am thinking about the type checking.  
How to check for a

```julia
julia> typeof(df.y)
Array{Union{Missing, Float64},1}

```

By the way, for MWE:

```julia
julia> df=DataFrame(x=[1,2,3],y=[1.0,missing,2.0])

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [December 8, 2019, 9:03pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/3 "2019-12-08T21:03:50Z")

</div>

ok, I think, it lacks elegance:

```julia
julia> df=DataFrame(x=[1,missing,3],y=[1.0,missing,2.0])
3×2 DataFrame
│ Row │ x │ y │
│ │ Int64 │ Float64 │
├─────┼─────────┼──────────┤
│ 1 │ 1 │ 1.0 │
│ 2 │ missing │ missing │
│ 3 │ 3 │ 2.0 │

julia> for col in names(df)
           if typeof(df[:,col])==Array{Union{Missing, Float64},1}
               df[:,col]= [ismissing(x) ? 0.0 : x for x in df[:,col] ]
           end
       end

julia> df
3×2 DataFrame
│ Row │ x │ y │
│ │ Int64 │ Float64 │
├─────┼─────────┼──────────┤
│ 1 │ 1 │ 1.0 │
│ 2 │ missing │ 0.0 │
│ 3 │ 3 │ 2.0 │

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 8, 2019, 9:09pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/4 "2019-12-08T21:09:32Z")

</div>

Check out the documentation for this in the DataFrames [documentation](http://juliadata.github.io/DataFrames.jl/stable/man/missing/).

It looks like the answer you want is `coalesce.(df, 0.0)`

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [December 8, 2019, 9:42pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/5 "2019-12-08T21:42:40Z")

</div>

Thanks for your suggestions!  
But I still get a warning:

```julia
┌ Warning: `setindex!(df::DataFrame, v::AbstractVector, ::Colon, col_ind::ColumnIndex)` is deprecated, use `begin
│ df[!, col_ind] = v
│ df
│ end` instead.

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 8, 2019, 10:03pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/6 "2019-12-08T22:03:02Z")

</div>

just use `df[!, col] = [ismissing(x) ? 0.0 ...`. Alternatively if you upgrade to the most recent version of DataFrames (released today), you that will work without a warning message.

---

<div class="post-metadata">

**Author:** ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)\
**Post date:** [December 8, 2019, 10:54pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/7 "2019-12-08T22:54:15Z")

</div>

The comments above are better as exact solutions to OP’s question, but I thought some people may find it interesting to know how to do this with MLJ:

```julia
using MLJ, DataFrames
df = DataFrame((x=[randn(5)..., missing], y=[randn(5)...,missing]))

filler = machine(FillImputer(continuous_fill=m->0), df)
fit!(filler)
dff = transform(filler, df)

```

The fill imputer can use other filling rules like the mean of the column or its median. The keyword `continuous_fill` takes a function to apply to “continuous” features as opposed to `count_fill` / `finite_fill` which take functions for count-data / categorical data.

---

<div class="post-metadata">

**Author:** ![Sijun](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@Sijun](https://discourse.julialang.org/u/Sijun)\
**Post date:** [December 9, 2019, 12:57am UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/8 "2019-12-09T00:57:09Z")

</div>

How about this one:

`mapcols(c->eltype(c)<:Union{Float64, Missing} ? coalesce.(c, 0.0) : c, df)`

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [December 9, 2019, 8:13am UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/9 "2019-12-09T08:13:28Z")

</div>

Is there a way to get all types which are part of Union?  
E.g.:

```julia
allTypes( Union{Float64, Missing} )
Float64
Missing

```

Or something similar, like e.g.

```julia
isInUnion(Float64,Union{Float64, Missing})
true

```

---

<div class="post-metadata">

**Author:** ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)\
**Post date:** [December 9, 2019, 8:27am UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/10 "2019-12-09T08:27:52Z")

</div>

A bit outside the scope of the question but `Float64 <: Union{Float64, Missing}` does the latter. See also [this post](https://discourse.julialang.org/t/get-non-missing-type-in-the-case-of-parametric-type/29109/4) on retrieving the non-missing type and corresponding pointer to [Missings.jl](https://github.com/JuliaData/Missings.jl).

---

<div class="post-metadata">

**Author:** ![simeonschaub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simeonschaub/32/216566_2.png) [@simeonschaub](https://discourse.julialang.org/u/simeonschaub)\
**Post date:** [December 9, 2019, 2:35pm UTC](https://discourse.julialang.org/t/replace-missing-with-0-0-in-dataframe/32029/11 "2019-12-09T14:35:29Z")

</div>

For the first part, `Base.uniontypes` should do exactly what you want.