# Dropping missing values in dataframes
**URL:** https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967
**Category:** General Usage
**Tags:** dataframes
**Created:** [October 25, 2020, 8:00am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967 "2020-10-25T08:00:28Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 8:00am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/1 "2020-10-25T08:00:29Z")
The dataframe column’s describe shows:
5 age 29.6991 0.42 28.0 80.0 177 Union{Missing, Float64}
I can get rid of the missing values with this:
replace!(train\_df.age, missing =\> median(skipmissing(train\_df[:age])));
However, the type is still not correct:
5 age 29.3616 0.42 28.0 80.0 0 Union{Missing, Float64}
How do I do this so that the type becomes Float64?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 25, 2020, 8:34am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/2 "2020-10-25T08:34:10Z")
Since you are modifying in-place, of course the element type of the container will remain the same. There are many solutions, eg
```julia
using DataFrames, StatsBase, Statistics
df = DataFrame(:age => Vector{Union{Float64,Missing}}(rand(100)))
df.age[sample(axes(df, 1), 10)] .= missing
df.age = replace(df.age, missing => median(skipmissing(df.age)))
```
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 9:06am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/3 "2020-10-25T09:06:10Z")
Yes, indeed that works. I didn’t know that those two versions of replace work so differently. Thanks!
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [October 25, 2020, 9:48am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/4 "2020-10-25T09:48:37Z")
I think you’re looking for [`disallowmissing!`](https://juliadata.github.io/DataFrames.jl/stable/lib/functions/#DataFrames.disallowmissing!)
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 10:24am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/5 "2020-10-25T10:24:35Z")
No. I’m looking for replacing missing values with reasonable guestimates.
[https://cran.r-project.org/web/views/MissingData.html](https://cran.r-project.org/web/views/MissingData.html)
---
### Author:  [@danielw2904](https://discourse.julialang.org/u/danielw2904)
#### Post date: [October 25, 2020, 11:56am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/6 "2020-10-25T11:56:57Z")
Maybe this can help [https://github.com/invenia/Impute.jl](https://github.com/invenia/Impute.jl)
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [October 25, 2020, 2:07pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/7 "2020-10-25T14:07:04Z")
Apologies, in this case I misunderstood - the title of the thread is “Dropping missing values in dataframes”, which I read as implying that you’re looking to drop the missing values.
In your OP you also mentioned that after replacing missing values “the type is still not correct”, showing an example of a vector of type `Union{Missing, Float64}` and asked explicitly about how you can make the type `Float64`, which is what `disallowmissing!` can help you do.
Imputation is somewhat orthogonal to this - after imputation you might still have a `Union` type, and therefore might still want to call `disallowmissing!` to simplify subsequent analysis steps.
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 2:34pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/8 "2020-10-25T14:34:43Z")
I’m sorry that my questions sound a bit confusing. I’ve been trying to figure out how to work with dataframes using a pipe of some kind, could be @linq like below, or could be one of the other ones.
```julia
using DataFrames
using DataFramesMeta
using CSV
using Statistics
data = "PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked\n87,0,3,\"Ford, Mr. William Neal\",male,16.0,1,3,W./C. 6608,34.375,,S\n89,1,1,\"Fortune, Miss. Mabel Helen\",female,23.0,3,2,19950,263.0,C23 C25 C27,S\n371,1,1,\"Harder, Mr. George Achilles\",male,25.0,1,0,11765,55.4417,E50,C\n421,0,3,\"Gheorgheff, Mr. Stanio\",male,,0,0,349254,7.8958,,C\n498,0,3,\"Shellard, Mr. Frederick William\",male,,0,0,C.A. 6212,15.1,,S\n511,1,3,\"Daly, Mr. Eugene Patrick\",male,29.0,0,0,382651,7.75,,Q\n538,1,1,\"LeRoy, Miss. Bertha\",female,30.0,0,0,PC 17761,106.425,,C\n627,0,2,\"Kirkland, Rev. Charles Leonard\",male,57.0,0,0,219533,12.35,,Q\n781,1,3,\"Ayoub, Miss. Banoura\",female,13.0,0,0,2687,7.2292,,C\n855,0,2,\"Carter, Mrs. Ernest Courtenay (Lilian Hughes)\",female,44.0,1,0,244252,26.0,,S\n"
df = CSV.read(IOBuffer(data))
df = @linq df |>
deletecols([:Name, :SibSp, :Parch, :Cabin, :Embarked, :Ticket, :Fare]) |>
rename(:PassengerId => :id, :Survived => :survived, :Pclass => :class, :Sex => :sex, :Age => :age)
df.age = replace(df.age, missing => median(skipmissing(df[:age])))
df = @transform(df, survived = convert.(Bool, :survived))
df.sex = categorical(df.sex)
df.class = categorical(df.class)
df
```
The code above works, but ideally those transformations would be in one pipe, and I couldn’t make it work. There is no particular need for that, except that it would look nice. Well, even with that it would not still be the best possible.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [October 25, 2020, 2:38pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/9 "2020-10-25T14:38:15Z")
Sorry, why can’t you put the `replace` inside `@transform`? That can all be done in one pipe.
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 2:39pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/10 "2020-10-25T14:39:11Z")
What do you mean?
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [October 25, 2020, 2:41pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/11 "2020-10-25T14:41:26Z")
> [@StatisticalMouse](#):
>
> ```julia
> df.age = replace(df.age, missing => median(skipmissing(df[:age])))
>
> ```
This can be wrtten as
```julia
@transform(df, age = replace(:age, missing => median(skipmissing(:age))
```
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 3:25pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/12 "2020-10-25T15:25:03Z")
Ok, I got it. This works:
```julia
using DataFrames
using DataFramesMeta
using CSV
using Statistics
data = "PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked\n87,0,3,\"Ford, Mr. William Neal\",male,16.0,1,3,W./C. 6608,34.375,,S\n89,1,1,\"Fortune, Miss. Mabel Helen\",female,23.0,3,2,19950,263.0,C23 C25 C27,S\n371,1,1,\"Harder, Mr. George Achilles\",male,25.0,1,0,11765,55.4417,E50,C\n421,0,3,\"Gheorgheff, Mr. Stanio\",male,,0,0,349254,7.8958,,C\n498,0,3,\"Shellard, Mr. Frederick William\",male,,0,0,C.A. 6212,15.1,,S\n511,1,3,\"Daly, Mr. Eugene Patrick\",male,29.0,0,0,382651,7.75,,Q\n538,1,1,\"LeRoy, Miss. Bertha\",female,30.0,0,0,PC 17761,106.425,,C\n627,0,2,\"Kirkland, Rev. Charles Leonard\",male,57.0,0,0,219533,12.35,,Q\n781,1,3,\"Ayoub, Miss. Banoura\",female,13.0,0,0,2687,7.2292,,C\n855,0,2,\"Carter, Mrs. Ernest Courtenay (Lilian Hughes)\",female,44.0,1,0,244252,26.0,,S\n"
df = CSV.read(IOBuffer(data))
df = @linq df |>
deletecols([:Name, :SibSp, :Parch, :Cabin, :Embarked, :Ticket, :Fare]) |>
rename(:PassengerId => :id, :Survived => :survived, :Pclass => :class, :Sex => :sex, :Age => :age) |>
transform(age = replace(:age, missing => median(skipmissing(:age)))) |>
transform(survived = convert.(Bool, :survived)) |>
transform(sex = categorical(:sex)) |>
transform(class = categorical(:class))
```
I find the documentation for this quite confusing. It’s in small pieces, while I would benefit more on having slightly larger examples like this.
---
### Author:  [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)
#### Post date: [October 25, 2020, 3:32pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/13 "2020-10-25T15:32:18Z")
What I would normally do is to try find examples of usage from Google. I did it here, but what I found were more small pieces that didn’t really help.
Here’s a radical idea: what if the documentation would already contain the examples that people otherwise need to ask in StackOverflow?
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [October 25, 2020, 3:38pm UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/14 "2020-10-25T15:38:07Z")
yes the DataFramesMeta docs certainly need an overhaul, and they will get one soon. The package is currently undergoing rapid development in preparation for a major release, and documentation will be part of that.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 26, 2020, 9:13am UTC](https://discourse.julialang.org/t/dropping-missing-values-in-dataframes/48967/15 "2020-10-26T09:13:39Z")
> [@StatisticalMouse](#):
>
> what if the documentation would already contain the examples that people otherwise need to ask in StackOverflow?
It would be way too long, few people would read it, and users would still complain that it is missing some example someone asked on SO.
The purpose of good documentation is to introduce building blocks that allow you to craft your own solution after some investment into understanding. Examples are needed to help with that.