# Combine more DataFrames with some columns missing

**URL:** <https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415>\
**Category:** New to Julia\
**Tags:** json, dataframes\
**Created:** [May 13, 2020, 5:53pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415 "2020-05-13T17:53:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![stej](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@stej](https://discourse.julialang.org/u/stej)\
**Post date:** [May 13, 2020, 5:53pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415/1 "2020-05-13T17:53:18Z")

</div>

Hi all,

I deal with API, that returns sometimes _incomplete_ JSON. Some properties are missing. That could be reproduced with this code:

```julia
using Query
using CSV
using JSON
using DataFrames
testjson = """
[{"A":5, "B":6}, {"A":7, "D":8}]
"""
js = JSON.parse(testjson)
df = DataFrame.(js)
vcat(df...)

```

Note that first object contains properties `A`, `B`, second `A`, `D`. As a result I’d like to have a `DataFrame` with 2 rows and columns A, B, D.

The `vcat` returns error, that is not surprising.

```julia
julia> vcat(df...)
ERROR: ArgumentError: column(s) D are missing from argument(s) 1, and column(s) B are missing from argument(s) 2
Stacktrace:
 [1] _vcat(::Array{DataFrame,1}; cols::Symbol) at C:\Users\u\.julia\packages\DataFrames\3ZmR2\src\abstractdataframe\abstractdataframe.jl:1421

```

**My “solution” looks quite ugly. Is there any other way how to do that?**

```julia
using Query
using CSV
using JSON
using DataFrames
testjson = """
[{"A":5, "B":6}, {"A":7, "D":8}]
"""
js = JSON.parse(testjson)
df = DataFrame.(js)

dfs = DataFrame.(df);
# get all column names through all JSON objects (= A, B, D)
dfcols = collect.(keys.(js)) |> 
              Iterators.flatten |> 
              @groupby(_) |> 
              @map(Name=key(_)) |> 
              collect
# for each DataFrame find missing columns and place some default value
for df = dfs
  missingcolumns = setdiff(dfcols, names(df))
  for col = missingcolumns
    df[!, col] .= ""
  end
end
# finally possible to concat the dataframes
dffinal = vcat(dfs...)

```

The result looks like this:

```julia
julia> dffinal
2×3 DataFrame
│ Row │ A │ B │ D │
│ │ Int64 │ Any │ Any │
├─────┼───────┼─────┼─────┤
│ 1 │ 5 │ 6 │ │
│ 2 │ 7 │ │ 8 │

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 13, 2020, 5:57pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415/2 "2020-05-13T17:57:02Z")

</div>

`cols = :union` in `vcat` should work, right?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 13, 2020, 5:58pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415/3 "2020-05-13T17:58:02Z")

</div>

if you don’t know, already, you can type `? vcat` in the REPL and pull up documentation for that function. Then you can read through it to find the documentation for data frames, and you would see that there is a keyword argument for what you want.

---

<div class="post-metadata">

**Author:** ![stej](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@stej](https://discourse.julialang.org/u/stej)\
**Post date:** [May 13, 2020, 7:04pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415/4 "2020-05-13T19:04:56Z")

</div>

Pretty! Thanks a lot.

It didn’t came to my mind that `vcat` might solve that. Honestly I found `vcat` just on StackOverflow when trying to combine more `DataFrame`s together.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 13, 2020, 8:07pm UTC](https://discourse.julialang.org/t/combine-more-dataframes-with-some-columns-missing/39415/5 "2020-05-13T20:07:09Z")

</div>

Please get into the habit of looking at `? fun` whenever you are confused about a function. It will probably have the answer.