# Transform! changes the type of my variable

**URL:** <https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968>\
**Category:** General Usage\
**Created:** [February 29, 2024, 5:50pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968 "2024-02-29T17:50:40Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [February 29, 2024, 5:50pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/1 "2024-02-29T17:50:40Z")

</div>

I have a mix of strings and missing values, so I have defined my DF column to be `Union{String,Missing}`

In rare cases, all the values in this column might be missing. In this case, `transform!` will convert the column to be `Missing` instead, dropping the ‘Union’.

For example:

```julia
using DataFrames
df1 = DataFrame(str=Union{String,Missing}[])
append!(df1.str, ["Hello", "World"])
println("df1 before : ", typeof(df1.str))
transform!(df1, names(df1) .=> ByRow(x -> ismissing(x) ? missing : isa(x, String) ? string(strip(x)) : x) .=> names(df1))
println("df1 after : ", typeof(df1.str))

df2 = DataFrame(str=Union{String,Missing}[])
append!(df2.str, [missing, missing])
println("df2 before : ", typeof(df2.str))
transform!(df2, names(df2) .=> ByRow(x -> ismissing(x) ? missing : isa(x, String) ? string(strip(x)) : x) .=> names(df2))
println("df2 after : ", typeof(df2.str))

```

gives methe following results:

```julia
df1 before : Vector{Union{Missing, String}}
df1 after : Vector{String}
df2 before : Vector{Union{Missing, String}}
df2 after : Vector{Missing}

```

I don’t want the type of the variable in my DF to change - it screws up subsequent processing in this rare case of all `missing` data. How can I avoid this?

Thanks

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [February 29, 2024, 6:33pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/2 "2024-02-29T18:33:56Z")

</div>

`allowmissing!(df2)` restores the types to `Union{Missing,String}` (with the optional column args restricting the operation to the columns modified by `transform!`).

See [Missing Data · DataFrames.jl](https://dataframes.juliadata.org/stable/man/missing/)

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [February 29, 2024, 7:11pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/3 "2024-02-29T19:11:51Z")

</div>

Thank you for this suggestion @hendri54 . However, it doesn’t work in my example. `df2` has been changed to type `Missing` by the transform. I need the `String` part of the union restored, not the `Missing` part.

In other words, `allowmissing!` achieves this:

```julia
df1 before : Vector{Union{Missing, String}}
df1 after : Vector{Union{Missing, String}}
df2 before : Vector{Union{Missing, String}}
df2 after : Vector{Missing}

```

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [February 29, 2024, 7:19pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/4 "2024-02-29T19:19:56Z")

</div>

To illustrate my issue further:

```julia
df2 = DataFrame(str=Union{String,Missing}[missing, missing])
transform!(df2, names(df2) .=> ByRow(x -> ismissing(x) ? missing : isa(x, String) ? string(strip(x)) : x) .=> names(df2))

df2[1, :str] = "Hello"

```

The final assignment fails:

```julia
ERROR: LoadError: cannot convert a value to missing for assignment

```

---

<div class="post-metadata">

**Author:** ![Nathan\_Boyer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nathan_boyer/32/14825_2.png) [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)\
**Post date:** [February 29, 2024, 7:27pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/5 "2024-02-29T19:27:32Z")

</div>

Works, but not pretty.

```julia
foreach(
    x -> df2[!, x] = convert(Vector{Union{String, Missing}}, df2[!, x]),
    names(df2)
)

```

or

```julia
transform!(
    df2,
    names(df2) .=> (x -> convert(Vector{Union{String,Missing}}, x)) .=> names(df2)
)

```

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [February 29, 2024, 7:38pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/6 "2024-02-29T19:38:37Z")

</div>

> [@Nathan\_Boyer](#):
>
> ```julia
> foreach(
> x -> df2[!, x] = convert(Vector{Union{String, Missing}}, df2[!, x]),
> names(df2)
> )
> 
> ```

This is good, but not all my columns are String types. I’ll need to iterate over just those columns that were initially `Union{String, Missing}` to restore the union. I only need to apply the transform to these anyway.

Thank you.

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [February 29, 2024, 7:52pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/7 "2024-02-29T19:52:09Z")

</div>

Even if it’s not pretty, this works now:

```julia
df2 = DataFrame(str=Union{String,Missing}[missing, missing])
for col in eachcol(df2)
    if col isa Union{String,Missing}
        transform!(df2, col .=> ByRow(x -> ismissing(x) ? missing : string(strip(x)) .=> col))
        convert(Vector{Union{String,Missing}}, df2[!, col])
    end
end

```

Thanks.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [March 2, 2024, 8:36pm UTC](https://discourse.julialang.org/t/transform-changes-the-type-of-my-variable/110968/8 "2024-03-02T20:36:44Z")

</div>

This is definitely a pretty unfortunate issue and I’m not sure a great way to work around it, to be honest.

Interestingly, R infers the type of output as a vector `character` elements in the same scenario

```julia
r$> df = tibble(str = c(NA, NA))

r$> df2 = mutate(df, str = trimws(str))

r$> df2
# A tibble: 2 × 1
  str  
  <chr> # <------- See here
1 NA   
2 NA   

```

I think a Base dev’s opinion might be useful for this. Is there a convenient way to tell Julia to allocate an output vector based on inferred types (if there is type stability, I guess), and not the actual type that is created?

This would be a nice quality-of-life improvement for `missing`s, but I’m not sure how it would all work

OP you may also find BangBang.jl’s `setindex!!` function useful. See [here](https://juliafolds.github.io/BangBang.jl/stable/#BangBang.setindex!!).