# Detecting missing in DataFrame columns

**URL:** <https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853>\
**Category:** New to Julia\
**Created:** [September 12, 2018, 11:59am UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853 "2018-09-12T11:59:22Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 11:59am UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/1 "2018-09-12T11:59:22Z")

</div>

I am still trying to get used to some of the syntax so I apologize in advance if this is a simple answer. I have a couple of columns in a DataFrame, some of which have a missing value and some columns do not. I am looking for a way to detect which columns have any missing value and then a way to see if any cells across the DataFrame have a single missing value.

Here is the code I have, I tried the two lines at the end and both come back all false even when there are missing values:

```julia
using Pkg
using DataFrames

df = DataFrame([5:6, [1, missing] ,[2, missing]])

colwise(ismissing, df)
colwise(ismissing, df[2,:])

```

---

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [September 12, 2018, 12:26pm UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/2 "2018-09-12T12:26:34Z")

</div>

```julia
julia> colwise(x -> any(ismissing.(x)), df)

3-element Array{Bool,1}:
 false
  true
  true

```

`ismissing` just asks if the column itself is a missing type, which it is not. Instead, you want to broadcast `ismissing` over the column data, and return `true` if `any` are missing.

**EDIT** : for your second question, to get the number of missing values in a column you can use

```julia
julia> colwise(x -> sum(ismissing.(x)), df)

3-element Array{Int64,1}:
 0
 1
 1

```

---

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 12:31pm UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/3 "2018-09-12T12:31:23Z")

</div>

Ah that makes perfect sense. I need to get used to thinking about whether I want to broadcast a function or not. I love the general flexibility of the language that way, just need to practice more.

I appreciate the help!

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 12, 2018, 2:23pm UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/4 "2018-09-12T14:23:28Z")

</div>

`describe(df)` will also show you whether or not it contains missing values, as well as the number of missings. Check out `?describe` for a better description.

---

<div class="post-metadata">

**Author:** ![Javier\_Quintero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/javier_quintero/32/10082_2.png) [@Javier\_Quintero](https://discourse.julialang.org/u/Javier_Quintero)\
**Post date:** [April 6, 2021, 9:01am UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/5 "2021-04-06T09:01:38Z")

</div>

colwise function is deprecated [Future of colwise #1595](https://github.com/JuliaData/DataFrames.jl/issues/1595)

Best option is `describe(df)` or use `eachcol`, like this:

`collect(any(ismissing.(c)) for c in eachcol(C))`  
or  
`collect(any(ismissing, c) for c in eachcol(C))`

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [April 6, 2021, 9:26am UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/6 "2021-04-06T09:26:55Z")

</div>

Using `mapcols` is also quite nice I think:

```julia
julia> mapcols(x -> any(ismissing, x), df)
1×3 DataFrame
 Row │ x1 x2 x3   
     │ Bool Bool Bool 
─────┼───────────────────
   1 │ false true true

```

or

```julia
julia> mapcols(Base.Fix1(any, ismissing), df)
1×3 DataFrame
 Row │ x1 x2 x3   
     │ Bool Bool Bool 
─────┼───────────────────
   1 │ false true true

```

---

<div class="post-metadata">

**Author:** ![Javier\_Quintero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/javier_quintero/32/10082_2.png) [@Javier\_Quintero](https://discourse.julialang.org/u/Javier_Quintero)\
**Post date:** [April 6, 2021, 1:14pm UTC](https://discourse.julialang.org/t/detecting-missing-in-dataframe-columns/14853/7 "2021-04-06T13:14:03Z")

</div>

Interesting mapcols returns a DataFrame like describe, and nice way to use any.