# Applying function to DataFrame with missing values in columns?

**URL:** <https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700>\
**Category:** Data\
**Tags:** question, dataframes\
**Created:** [September 8, 2018, 10:04am UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700 "2018-09-08T10:04:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![onetonfoot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/onetonfoot/32/2697_2.png) [@onetonfoot](https://discourse.julialang.org/u/onetonfoot)\
**Post date:** [September 8, 2018, 10:04am UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700/1 "2018-09-08T10:04:00Z")

</div>

Let say I’ve got a data frame like this:

```julia
df = DataFrame(:x => [1 ,3, missing], :y => ["Some ascii text" ,"Some chinese text 恒基",missing])

```

In simple cases, `map` seems to do what I want and just ignores/returns the missing values, for example the following doesn’t error.

```julia
g(x) = x* 10
map(g,df[:x])

Union{Missing, Int64}[10, 30, missing]

```

However in other cases it doesn’t seem to work, like this will chuck an error.

```julia
replace_non_ascii(c) = isascii(c) ? c : ' '
f(x) = map(replace_non_ascii,x)
map(f,df[:y])

```

I can get around it by defining the function again `f(x::Missing) = missing`, but was wondering if theres is a nicer way?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 8, 2018, 10:45am UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700/2 "2018-09-08T10:45:00Z")

</div>

Currently, there is a broad set of functions pre-defined to handle `missing` values. As you’ve seen though, there are also functions that are missing “unaware”. There’s a proposal [here](https://github.com/JuliaData/Missings.jl/pull/89) to add a generic “lifting” function that would allow any function to propagate missing. The idea is that we can try out that functionality in a package and potentially add support for that to the language itself in the form of an operator or something.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 8, 2018, 11:41pm UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700/3 "2018-09-08T23:41:13Z")

</div>

You can use [Query.jl](https://github.com/queryverse/Query.jl) for this:

```julia
df |> @map({y=f.(_.y)}) |> DataFrame

```

Note how the `f` function here is called with the `.` broadcasting syntax: that automatically lifts the function and makes it deal with `NA` values, even if `f` doesn’t know about `NA` values at all.

---

<div class="post-metadata">

**Author:** ![onetonfoot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/onetonfoot/32/2697_2.png) [@onetonfoot](https://discourse.julialang.org/u/onetonfoot)\
**Post date:** [September 9, 2018, 11:44am UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700/4 "2018-09-09T11:44:17Z")

</div>

Yeah I think having language level support for this kind of thing would be ideal, until then I’ll try make use of Query.jl. Cheers guys