# Expanding Named Tuples

**URL:** <https://discourse.julialang.org/t/expanding-named-tuples/62435>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [June 5, 2021, 1:40pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435 "2021-06-05T13:40:40Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [June 5, 2021, 1:40pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/1 "2021-06-05T13:40:40Z")

</div>

Starting from this situation (or similar),

```julia
julia> hcat(DataFrame(a=[1,2]), [(b=1,c=2), (b=3,c=4)])
2×2 DataFrame
 Row │ a x1             
     │ Int64 NamedTup…      
─────┼───────────────────────
   1 │ 1 (b = 1, c = 2)
   2 │ 2 (b = 3, c = 4)

```

is there a direct way to obtain this result?

In general, in what ways can a flat DataFrame be obtained, starting from such a situation?

```julia
2×3 DataFrame
 Row │ a b c     
     │ Int64 Int64 Int64 
─────┼─────────────────────
   1 │ 1 1 2
   2 │ 2 3 4

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [June 5, 2021, 1:49pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/2 "2021-06-05T13:49:08Z")

</div>

Not clear what the real _situation_ is, but this gives the desired result:

```julia
julia> hcat( DataFrame(a=[1,2]), DataFrame([(b=1,c=2),(b=3,c=4)]) )
2×3 DataFrame
 Row │ a b c
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 1 2
   2 │ 2 3 4

```

But of course, I have just changed the _situation_ to get this.  
What is your real starting point?

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [June 5, 2021, 2:03pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/3 "2021-06-05T14:03:23Z")

</div>

No real situation.  
I was reading [this PR](https://github.com/JuliaData/DataFrames.jl/pull/2777) and I asked myself the question.

I would expect a solution of the following type, (ie having as arguments the dataframe and the column to expand) but without hcat.

```julia
dfnt=hcat(DataFrame(a=[1,2]), [(b=1,c=2), (b=3,c=4)])
hcat(dfnt.a,DataFrame(dfnt.x1))

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 5, 2021, 2:09pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/4 "2021-06-05T14:09:59Z")

</div>

The problem (as explained in the PR you linked) is that we already define `hcat(df::AbstractDataFrame, v::AbstractVector)`. We would _like_ to add something along the lines of

```julia
function hcat(df::AbstractDataFrame, t::Any)
    if Tables.istable(t)
        hcat(df, DataFrame(t; copycols = false)
    end
end

```

But of course, we can’t use dispatch to check if something is a `Table` in the Tables.jl-sense. And there are many things that are both `<: Vector` and satisfy `Tables.istable`, like a vector of named tuples.

But we are post 1.0, so we can’t break `hcat(df::AbstractDataFrame, v::AbstractVector)`. So no, you won’t be able to do

```julia
hcat(DataFrame(a=[1,2]), [(b=1,c=2), (b=3,c=4)])

```

and have it automatically flatten. That would break a post-1.0 guarantee of stability.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [June 5, 2021, 2:27pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/5 "2021-06-05T14:27:42Z")

</div>

Hi @pdeffebach,

I’m not sure if I understand your answer correctly (my knowledge of Julia is very limited), but I would like to be sure that I have asked my question correctly and so I try to ask it again.

If after somehow transforming a dataframe (perhaps obtained by reading a JSON file !?) I get some columns that are vectors of named tuples, how can I expand/flat them to obtain distinct columns corresponding to the names of the keys of the named tuples?

PS  
I don’t intend to use / modify the hcat function

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 5, 2021, 2:49pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/6 "2021-06-05T14:49:50Z")

</div>

Calling `DataFrame(v)` should work.

If you have this scenario

```julia
julia> vnt = [(a = 1, b = (c = 2, d = 3)), (a = 4, b = (c = 5, d = 6))]
2-element Vector{NamedTuple{(:a, :b), Tuple{Int64, NamedTuple{(:c, :d), Tuple{Int64, Int64}}}}}:
 (a = 1, b = (c = 2, d = 3))
 (a = 4, b = (c = 5, d = 6))

```

then I’m not 100% what the solution is, but I’m sure other people can help out. Here is one solution with recursion

```julia
julia> vnt = [(a = 1, b = (c = 2, d = 3)), (a = 4, b = (c = 5, d = 6))];

julia> function unnest!(d, nt)
           for (n, v) in pairs(nt)
               if v isa NamedTuple
                   unnest!(d, v)
               else
                   push!(d, n => v)
               end
           end
       end;

julia> function unnest(nt)
           d = Dict{Symbol, Any}()
           unnest!(d, nt)
           return d
       end;

julia> Tables.istable(unnest.(vnt))
true

julia> DataFrame(unnest.(vnt))
2×3 DataFrame
 Row │ a d c     
     │ Int64 Int64 Int64 
─────┼─────────────────────
   1 │ 1 3 2
   2 │ 4 6 5

```

But this solution has some problems. In particular, it won’t have consistent column ordering (this can be fixed using an ordered dict from `OrderedCollections.jl`).

But I feel like we have good solutions to this problem that I am not finding at the moment.

EDIT: Also look at JSONTables.jl, for a particular JSON-oriented use-case

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [June 5, 2021, 3:48pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/7 "2021-06-05T15:48:56Z")

</div>

```julia
julia> t=hcat(DataFrame(a=[1,2]), [(b=1,c=2), (b=3,c=4)])
2×2 DataFrame
 Row │ a x1
     │ Int64 NamedTup…
─────┼───────────────────────
   1 │ 1 (b = 1, c = 2)
   2 │ 2 (b = 3, c = 4)

```

From a column like `x1` above, you can proceed with:

```julia
julia> DataFrame(t[!,:x1])
2×2 DataFrame
 Row │ b c
     │ Int64 Int64
─────┼──────────────
   1 │ 1 2
   2 │ 3 4

```

Just saying, perhaps that’s what you are looking for.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [June 5, 2021, 5:36pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/8 "2021-06-05T17:36:51Z")

</div>

No. This is not exactly what I am looking for, because I would like to have the whole dataframe as an output.

Something like this (which came to my mind trying to figure out the functions of @pdeffebach )

```julia
df=DataFrame(vnt)
ransform(df, :b=>ByRow(x->(x.c,x.d))=>[:c,:d])

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [June 5, 2021, 5:44pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/9 "2021-06-05T17:44:28Z")

</div>

Well, the whole DataFrame is than this:

```julia
julia> hcat(DataFrame(a=t[!,:a]),DataFrame(t[!,:x1]))
2×3 DataFrame
 Row │ a b c
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 1 2
   2 │ 2 3 4

```

But, never mind, it seems I am missing the point. I also didn’t read the PR you linked.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 5, 2021, 6:37pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/10 "2021-06-05T18:37:25Z")

</div>

use `AsTable` as the output

```julia
df=DataFrame(vnt)
transform(df, :b=>ByRow(identity) => AsTable)

```

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [June 5, 2021, 6:42pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/11 "2021-06-05T18:42:24Z")

</div>

> [@pdeffebach](#):
>
> But of course, we can’t use dispatch to check if something is a `Table` in the Tables.jl-sense.

```julia
istableval(x) = Val(Tables.istable(x)) # const prop should make this infer correctly
hcat(df::AbstractDataFrame, t) = _hcat(istableval(t), df, t)
_hcat(::Val{true}, df, t) = hcat(df, DataFrame(t; copycols = false))
_hcat(::Val{false}, df, t) = ...

```

I placed the `Val` as the first argument of `_hcat` because `hcat` often accepts `Vararg`s.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 5, 2021, 6:48pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/12 "2021-06-05T18:48:21Z")

</div>

Fair enough! Perhaps this technique can be used internally in DataFrames more.

Still doesn’t get around the problem that things are both `<: AbstractVector` and tables.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [June 5, 2021, 7:22pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/13 "2021-06-05T19:22:35Z")

</div>

it seems that ByRow is not needed

```julia
select(transform(df, :b=>identity=>AsTable), Not(:b))

```

just out of curiosity, why doesn’t the following expression work like the previous one?

```julia
select(transform(df, :b=>AsTable), Not(:b))

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 5, 2021, 8:13pm UTC](https://discourse.julialang.org/t/expanding-named-tuples/62435/14 "2021-06-05T20:13:18Z")

</div>

I think it probably should work, and I’ve filed an issue [here](https://github.com/JuliaData/DataFrames.jl/issues/2779). But no guarantees because the mini-language is complicated enough as-is.