# How to apply replace to whole DataFrame like in Python?

**URL:** <https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797>\
**Category:** General Usage\
**Tags:** question, dataframes\
**Created:** [May 13, 2023, 3:27pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797 "2023-05-13T15:27:46Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hungpham3112](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hungpham3112/32/34315_2.png) [@hungpham3112](https://discourse.julialang.org/u/hungpham3112)\
**Post date:** [May 13, 2023, 3:27pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/1 "2023-05-13T15:27:47Z")

</div>

To replace a character in df in Python we can:

```jl
df.replace("?", np.nan, inplace = True)

```

but it seems Julia not support replace to the whole DataFrame

```jl
replace(df, "?" => missing)

```

returns:

> MethodError: no method matching similar(::DataFrames.DataFrame, ::Type{Any})

After some testing, I found the alternative using for loop:

```jl
for col in names(df)
    replace!(df[!, col], "?" => missing)
end

```

but actually I prefer one-liner code than for loop.

How can I achieve the same behavior like python code with Julia? What is the cleanest way to do that? Thank you.

Edit1:  
Here is sample df:

```julia
df = DataFrame(a = [1, 2.2, "?"], b = ["b", "a" , "?"])

```

my dataframe that I’m working on is not only contain `String` but also `Float` and `Int`

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 13, 2023, 3:40pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/2 "2023-05-13T15:40:44Z")

</div>

Edit: This is unfortunately a false friend solution which only appears to work upon casual inspection.

~~You can use broadcasting to apply it to all values:~~

> ****
>
> ```julia df = DataFrame(a = ["a", "b", "?"], b = ["?", "b", "?"]) 3×2 DataFrame Row │ a b │ String String ─────┼──────────────── 1 │ a ? 2 │ b b 3 │ ? ?
> 
> replace.(df, “?” =\> missing)  
> 3×2 DataFrame  
> Row │ a b  
> │ String String  
> ─────┼──────────────────  
> 1 │ a missing  
> 2 │ b b  
> 3 │ missing missing
> 
> ```julia
> 
> ~~It does not seem to work with `replace!` though :frowning:~~
> <\details>
> ```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [May 13, 2023, 3:45pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/3 "2023-05-13T15:45:57Z")

</div>

in-place operation requires that your original columns accept `missing`:

```julia
julia> df = DataFrame(a = ["a", "b", "?"], b = ["?", "b", "?"])
3×2 DataFrame
 Row │ a b
     │ String String
─────┼────────────────
   1 │ a ?
   2 │ b b
   3 │ ? ?

julia> allowmissing!(df) # this is needed for in-place to work
3×2 DataFrame
 Row │ a b
     │ String? String?
─────┼──────────────────
   1 │ a ?
   2 │ b b
   3 │ ? ?

julia> @. df = ifelse(df == "?", missing, df)
3×2 DataFrame
 Row │ a b
     │ String? String?
─────┼──────────────────
   1 │ a missing
   2 │ b b
   3 │ missing missing

```

Otherwise you need to replace columns:

```julia
julia> df = DataFrame(a = ["a", "b", "?"], b = ["?", "b", "?"])
3×2 DataFrame
 Row │ a b
     │ String String
─────┼────────────────
   1 │ a ?
   2 │ b b
   3 │ ? ?

julia> df[!, :] = @. ifelse(df == "?", missing, df)
3×2 DataFrame
 Row │ a b
     │ String? String?
─────┼──────────────────
   1 │ a missing
   2 │ b b
   3 │ missing missing

```

Note that it is not the same as:

```julia
julia> df = @. ifelse(df == "?", missing, df)
3×2 DataFrame
 Row │ a b
     │ String? String?
─────┼──────────────────
   1 │ a missing
   2 │ b b
   3 │ missing missing

```

as this allocates a new data frame, while the above writes new columns into an existing data frame.

---

<div class="post-metadata">

**Author:** ![hungpham3112](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hungpham3112/32/34315_2.png) [@hungpham3112](https://discourse.julialang.org/u/hungpham3112)\
**Post date:** [May 13, 2023, 4:19pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/5 "2023-05-13T16:19:53Z")

</div>

1. Can you explain the details behind error `MethodError: no method matching similar(::DataFrames.DataFrame, ::Type{Any})`, it doesn’t make sense to me.  
Does `replace() use `similar()` under the hood?
2. Why do you use ifelse() instead of replace() in this situation?

---

<div class="post-metadata">

**Author:** ![hungpham3112](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hungpham3112/32/34315_2.png) [@hungpham3112](https://discourse.julialang.org/u/hungpham3112)\
**Post date:** [May 13, 2023, 4:31pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/6 "2023-05-13T16:31:49Z")

</div>

Idk why Julia behavior is so weird.  
AFAIK, the dataframe after changing should be

```julia
replace.(df, "?" => missing)
3×2 DataFrame
 Row │ a b       
     │ String? String?
─────┼──────────────────
   1 │ a missing 
   2 │ b b       
   3 │ missing missing 

```

not

```julia
replace.(df, "?" => missing)
3×2 DataFrame
 Row │ a b       
     │ String String
─────┼──────────────────
   1 │ a missing 
   2 │ b b       
   3 │ missing missing 

```

because “?” =\> missing not “?” =\> “missing”

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/9/f912e8ef6ec193909042385162c31477ea412203.png)

It doesn’t make sense to me. Maybe I’m lacking knowledge about type system in Julia or maybe Julia is designed like that.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [May 13, 2023, 4:42pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/7 "2023-05-13T16:42:50Z")

</div>

@bkamins Explained above: the original column is of type `String`, so when you try to add `missing`, it is converted to a `String`. That’s why he suggested the `allowmissing!` method to change the type of the columns to accept `missing`.

---

<div class="post-metadata">

**Author:** ![tiZ](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tiz/32/211800_2.png) [@tiZ](https://discourse.julialang.org/u/tiZ)\
**Post date:** [May 13, 2023, 5:01pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/8 "2023-05-13T17:01:12Z")

</div>

I have a question here for why not just use @DrChainsaw solution?  
it looks simple and easy to understand.

```julia
df = DataFrame(a = ["a", "b", "?"], b = ["?", "b", "?"])
df = replace.(df, "?" => missing)

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [May 13, 2023, 5:05pm UTC](https://discourse.julialang.org/t/how-to-apply-replace-to-whole-dataframe-like-in-python/98797/9 "2023-05-13T17:05:07Z")

</div>

> [@tiZ](#):
>
> why not just use @DrChainsaw solution?

For two reasons:

1. it allocates a new data frame, it is not in-place.
2. it converts `missing` to `"missing"`.

The point is that `replace` function has the following contract:

```julia
replace(s::AbstractString, pat=>r, [pat2=>r2, ...]; [count::Integer])

```

so essentially what you are doing is:

```julia
julia> replace("?", "?" => missing)
"missing"

```

To see that this is the case see:

```julia
julia> df = DataFrame(x = ["am I ?", "? or else", "now ? in the middle"])
3×1 DataFrame
 Row │ x
     │ String
─────┼─────────────────────
   1 │ am I ?
   2 │ ? or else
   3 │ now ? in the middle

julia> replace.(df, "?" => missing)
3×1 DataFrame
 Row │ x
     │ String
─────┼───────────────────────────
   1 │ am I missing
   2 │ missing or else
   3 │ now missing in the middle

```

In short - in this case `replace` does something completely different than you think it does.