# (DataFrames.jl) Initially allowing a DataFrame to receive missing

**URL:** <https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [December 2, 2021, 8:44am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436 "2021-12-02T08:44:35Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [December 2, 2021, 8:44am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/1 "2021-12-02T08:44:36Z")

</div>

According to [`allowmissing!` in DataFrames.jl](https://dataframes.juliadata.org/stable/lib/functions/#DataFrames.allowmissing!),  
it is possible to make a DataFrame “able to receive missing”:

```julia
julia> using DataFrames
d
julia> df = DataFrame()
0×0 DataFrame

julia> push!(df, (;a=1, b=1.0))
1×2 DataFrame
 Row │ a b
     │ Int64 Float64
─────┼────────────────
   1 │ 1 1.0

julia> push!(df, (;a=1, b=missing))
┌ Error: Error adding value to column :b.
└ @ DataFrames /Users/jinrae/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1483
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Float64
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/twiceprecision.jl:262
  ...
Stacktrace:
 [1] push!(a::Vector{Float64}, item::Missing)
   @ Base ./array.jl:994
 [2] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Missing}}; cols::Symbol, promote::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1465
 [3] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Missing}})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1353
 [4] top-level scope
   @ REPL[4]:1

julia> allowmissing!(df)
1×2 DataFrame
 Row │ a b
     │ Int64? Float64?
─────┼──────────────────
   1 │ 1 1.0

julia> push!(df, (;a=1, b=missing))
2×2 DataFrame
 Row │ a b
     │ Int64? Float64?
─────┼───────────────────
   1 │ 1 1.0
   2 │ 1 missing

```

However, I don’t get it how to initially allow a DataFrame receivable to missing data.  
For example, the following things didn’t work.

```julia
julia> using DataFrames

julia> df = DataFrame()
al0×0 DataFrame

julia> allowmissing!(df)
0×0 DataFrame

julia> push!(df, (;a=1, b=1.0))
pu1×2 DataFrame
 Row │ a b
     │ Int64 Float64
─────┼────────────────
   1 │ 1 1.0

julia> push!(df, (;a=1, b=missing))
┌ Error: Error adding value to column :b.
└ @ DataFrames /Users/jinrae/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1483
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Float64
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/base/twiceprecision.jl:262
  ...
Stacktrace:
 [1] push!(a::Vector{Float64}, item::Missing)
   @ Base ./array.jl:994
 [2] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Missing}}; cols::Symbol, promote::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1465
 [3] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Missing}})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1353
 [4] top-level scope
   @ REPL[5]:1

```

---

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [December 2, 2021, 8:48am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/2 "2021-12-02T08:48:50Z")

</div>

Although there is a remedy that pre-assigning columns’ type as

```julia
julia> df = DataFrame(a = Int[], b = Union{Missing,Float64}[])
0×2 DataFrame

julia> push!(df, (;a=1, b=1.0))
1×2 DataFrame
 Row │ a b
     │ Int64 Float64?
─────┼─────────────────
   1 │ 1 1.0

julia> push!(df, (;a=1, b=missing))
2×2 DataFrame
 Row │ a b
     │ Int64 Float64?
─────┼──────────────────
   1 │ 1 1.0
   2 │ 1 missing

```

But what if I don’t want to specify a type of each column? Or, I sometimes can’t even specify columns themselves.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 2, 2021, 9:10am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/3 "2021-12-02T09:10:31Z")

</div>

I don’t think what you’re after is possible - an empty DataFrame is actually empty, so can’t hold type information:

```julia
julia> df = DataFrame()
0×0 DataFrame

julia> getfield(df, :columns)
AbstractVector[]

```

so the columns only get created when you `push!` something. You can specify columns without types which will take `Any` value:

```julia
julia> df = DataFrame(a = [], b = [])
0×2 DataFrame

julia> getfield(df, :columns)
2-element Vector{AbstractVector}:
 Any[]
 Any[]

julia> push!(df, (; a = 5.0, b = 4))
1×2 DataFrame
 Row │ a b   
     │ Any Any 
─────┼──────────
   1 │ 5.0 4

julia> push!(df, (; a = 5.0, b = missing))
2×2 DataFrame
 Row │ a b       
     │ Any Any     
─────┼──────────────
   1 │ 5.0 4
   2 │ 5.0 missing 

```

but this is of course problematic from a performance perspective, and you still need to know the number of columns.

Why can’t you just do `allowmissing!(df)` after the first `push!`?

---

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [December 2, 2021, 9:39am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/4 "2021-12-02T09:39:34Z")

</div>

I thought about your suggestion, but what if there is a `missing` in the first `push!`?  
Then, the corresponding column’s type would be `Missing`, which does not allow concatenating other types.

For example,

```julia
julia> using DataFrames
df =
julia> df = DataFrame()
0×0 DataFrame

julia> push!(df, (;a=1, b=missing))
1×2 DataFrame
 Row │ a b
     │ Int64 Missing
─────┼────────────────
   1 │ 1 missing

julia> allowmissing!(df)
1×2 DataFrame
 Row │ a b
     │ Int64? Missing
─────┼─────────────────
   1 │ 1 missing

julia> push!(df, (;a=1, b=1.0))
┌ Error: Error adding value to column :b.
└ @ DataFrames /Users/jinrae/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1483
ERROR: MethodError: convert(::Type{Union{}}, ::Float64) is ambiguous. Candidates:
  convert(::Type{T}, x::Number) where T<:AbstractChar in Base at char.jl:184
  convert(::Type{T}, x::Number) where T<:Number in Base at number.jl:7
  convert(::Type{Union{}}, x) in Base at essentials.jl:216
  convert(::Type{T}, arg) where T<:VecElement in Base at baseext.jl:19
Possible fix, define
  convert(::Type{Union{}}, ::Number)
Stacktrace:
 [1] convert(#unused#::Type{Missing}, x::Float64)
   @ Base ./missing.jl:69
 [2] push!(a::Vector{Missing}, item::Float64)
   @ Base ./array.jl:994
 [3] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Float64}}; cols::Symbol, promote::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1465
 [4] push!(df::DataFrame, row::NamedTuple{(:a, :b), Tuple{Int64, Float64}})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:1353
 [5] top-level scope
   @ REPL[5]:1

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 2, 2021, 9:43am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/5 "2021-12-02T09:43:13Z")

</div>

I just find it hard to imagine a situation where I `push!` things repeatedly to a DataFrame and have no idea about the number of columns or there types?

Maybe a DataFrame isn’t the right thing for your use case, why not just a vector of vectors?

---

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [December 2, 2021, 10:04am UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/6 "2021-12-02T10:04:28Z")

</div>

Hmm…

You might be right.  
I need to consider my problem again.  
Thank you for your advice 🙂

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [December 2, 2021, 12:43pm UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/7 "2021-12-02T12:43:23Z")

</div>

An easy way to build a table row-by-row without specifying types upfront:

```julia
using BangBang
tbl = Union{}[]
tbl = push!!(tbl, (a=1, b=1.0))
tbl = push!!(tbl, (a=1, b=missing))
...

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 2, 2021, 12:51pm UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/8 "2021-12-02T12:51:22Z")

</div>

DataFrames.jl has this too

```julia
julia> df = DataFrame();

julia> push!(df, (a =1, b = 2))
1×2 DataFrame
 Row │ a b
     │ Int64 Int64
─────┼──────────────
   1 │ 1 2

julia> push!(df, (a =3, b = missing); promote = true)
2×2 DataFrame
 Row │ a b
     │ Int64 Int64?
─────┼────────────────
   1 │ 1 2
   2 │ 3 missing

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 2, 2021, 1:33pm UTC](https://discourse.julialang.org/t/dataframes-jl-initially-allowing-a-dataframe-to-receive-missing/72436/9 "2021-12-02T13:33:07Z")

</div>

Ah, that’s the solution then, as this also works in reverse (i.e. promote a `Missing` only column to the approriate union type)