# Remove / replace undef values in DataFrame

**URL:** <https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [October 5, 2021, 7:38am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219 "2021-10-05T07:38:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [October 5, 2021, 7:38am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/1 "2021-10-05T07:38:13Z")

</div>

Hi,  
Is there a way to replace all undef value in a Dataframe column with NA or another value?  
And how do I get rid of all the rows that have at least 1 undef value?  
Thank you

---

<div class="post-metadata">

**Author:** ![Karajan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/karajan/32/8545_2.png) [@Karajan](https://discourse.julialang.org/u/Karajan)\
**Post date:** [October 5, 2021, 7:59am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/2 "2021-10-05T07:59:42Z")

</div>

I think the first question should be how the `undef`s got there in the first place because that should never happen.

`DataFrames` also seems to prevent this:

```julia
julia> o = Vector{String}(undef, 10)
10-element Vector{String}:
 #undef
[...]

julia> p = DataFrame(o)
ERROR: UndefRefError: access to undefined reference
[...]

```

Could you provide a MWE (minimal working example)?

---

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [October 5, 2021, 8:05am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/3 "2021-10-05T08:05:17Z")

</div>

Here is a DataFrame that contains undef values:

```julia
df = DataFrame(
  A = Vector{String}(undef, 5),
  B = [5,1,2,undef,4]
)

```

---

<div class="post-metadata">

**Author:** ![Karajan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/karajan/32/8545_2.png) [@Karajan](https://discourse.julialang.org/u/Karajan)\
**Post date:** [October 5, 2021, 8:13am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/4 "2021-10-05T08:13:43Z")

</div>

Ah I see, thank you.

If you want to fill an array with values like

```julia
a = zeros(10)
for i in eachindex(a)
    a[i] = 5
end

```

you can avoid filling the array with zeros (`zeros(10)`) since you _know_ you are going to overwrite the values anyway. So here using

```julia
a = Vector{Float64}(undef, 10)
for i in eachindex(a)
    a[i] = 5
end

```

would save you a tiny amount of time.

However, this is pretty much the only time you should use `undef`. If you have code where you can access `undef`s, something went wrong. Since you seem to have control over the arrays themselves I would advise you to overwrite the `undefs` immediately or not use them in the first place.

What are you using them for? Maybe something like `missing` would be more fitting (and way easier and safer to work with)?

---

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [October 5, 2021, 8:17am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/5 "2021-10-05T08:17:46Z")

</div>

I did not build the DataFrame, I got it like this in a .jls file and I loaded the DataFrame with the following code:

```julia
df = Serialization.deserialize("data.jls")

```

---

<div class="post-metadata">

**Author:** ![Karajan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/karajan/32/8545_2.png) [@Karajan](https://discourse.julialang.org/u/Karajan)\
**Post date:** [October 5, 2021, 8:32am UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/6 "2021-10-05T08:32:43Z")

</div>

Well, it sounds like something went wrong with the deserialization (possibly due to “In general, this process will not work if the reading and writing are done by different versions of Julia, or an instance of Julia with a different system image.”?). For anything but short term saving probably CSV.jl, JDF.jl, etc. would be a better choice in the future.

If you have any way to access the original data that would probably the easiest option but if not I appreciate all that advice is not going to help you.

I hacked something together that can check if a single element is `undef`. Maybe someone else knows of a better way.

```julia
df = DataFrame(
  A = Vector{String}(undef, 5),
  B = [5,1,2,undef,4]
)
a = df.A
isassigned(a) # false
a[1] = "hi"
isassigned(a) # true
isassigned(Ref(@view a[1]).x) # true
isassigned(Ref(@view a[2]).x) # false
b = df.B
b[3] === UndefInitializer() # false
b[4] === UndefInitializer() # true

```

Using this I would go through all the data to replace all the `undef`s.

(PS: make extra sure the rest of your data is intact. It seems unlikely to me that something blew a few holes in your dataset but the rest was untouched 😉)

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [October 5, 2021, 12:27pm UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/7 "2021-10-05T12:27:28Z")

</div>

> [@gitboy16](#):
>
> `[5,1,2,undef,4]`

This does not seem like a realistic example to me. This is not how a `Vector{Int}` with undefined positions would look like at all. `[5,1,2,undef,4]` is a `Vector{Any}` with 3 `Int` and a `UndefInitializer` object that would never exist there unless something very wrong was done. If you create a `Vector{Int}` with `Vector{Int}(undef, 4)` it will look like:

```julia
4-element Array{Int64,1}:
 140534710323696
 140534774110064
 140534710323728
               0

```

It will _never_ have an `undef` inside it, because `Int` returns true for `isbitstype` and, therefore, each undefined position is just the `Int` value of the dirty bits from the memory allocated for the array. There is no way to represent `undef` with an `Int`.

---

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [October 5, 2021, 1:08pm UTC](https://discourse.julialang.org/t/remove-replace-undef-values-in-dataframe/69219/8 "2021-10-05T13:08:05Z")

</div>

I made up a DataFrame with undef. The DataFrame I use only have `String` columns and I can’t share it here.