# DataFrames.jl vcat column data from different rows

**URL:** <https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [August 16, 2022, 6:08am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809 "2022-08-16T06:08:06Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![ethomag](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ethomag/32/421_2.png) [@ethomag](https://discourse.julialang.org/u/ethomag)\
**Post date:** [August 16, 2022, 6:08am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/1 "2022-08-16T06:08:06Z")

</div>

I have a number of networks packets with sampled data, that is fragmented and was wondering whether de-fragmentation can be done with DataFrames or if I have to pre-process the data first ?

```julia
sort!(df, [:frame, :subframe, :seq, :sampleno])

467×6 DataFrame
 Row │ frame subframe seq sampleno nsamples samples
     │ UInt8 UInt16 UInt16 UInt32 UInt32 Array{Int8,
─────┼─────────────────────────────────────────────────────────────
   1 │ 4 7 13 0 330 [-1, 0, 144… 
   2 │ 4 7 13 330 16 [0, 0, 0, 0… 
   3 │ 4 8 0 0 112 [-1, 0, 144… 
   4 │ 4 8 0 112 234 [16, -118, … 
   5 │ 4 8 1 0 346 [-1, 0, 144… 

```

The rows with the same (`frame`, `subframe` and `seq`) are fragmented into several rows (packets).

So I want all groups of rows with the same (`frame`, `subframe` and `seq`) to be collapsed to one row with the data in each `samples` columns concatenated.

I have read the docs and watched some Tutorials, but could not find much with vector-valued column data. So I tried this

```julia
gd = groupby(df, [:frame, :subframe, :seq] )
dfm = combine(gd, :samples => vcat, :nsamples => sum)

```

But I get the same number of rows in the resulting DataFrame and the samples in `samples_vcat` are not concatenated. The only thing that worked as I expected is the `nsamples_sum` column, which seems to contain the sum of all `nsamples` in each group in `gd`.

Any help is much appreciated.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [August 16, 2022, 6:12am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/2 "2022-08-16T06:12:58Z")

</div>

```julia
julia> df = DataFrame(x = ["a", "a", "b", "b"], y = [[1,2],[3,4],[5,6],[7,8]])
4×2 DataFrame
 Row │ x y      
     │ String Array… 
─────┼────────────────
   1 │ a [1, 2]
   2 │ a [3, 4]
   3 │ b [5, 6]
   4 │ b [7, 8]

julia> combine(groupby(df, :x), :y => Ref ∘ (x -> reduce(vcat, x)) => :y)
2×2 DataFrame
 Row │ x y            
     │ String Array…       
─────┼──────────────────────
   1 │ a [1, 2, 3, 4]
   2 │ b [5, 6, 7, 8]

```

---

<div class="post-metadata">

**Author:** ![ethomag](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ethomag/32/421_2.png) [@ethomag](https://discourse.julialang.org/u/ethomag)\
**Post date:** [August 16, 2022, 6:24am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/3 "2022-08-16T06:24:42Z")

</div>

Many thanks @nilshg, worked like a charm! Could you elaborate on the `Ref ∘ ( ... )` ? Is this a standard trick you do with vector-valued columns ?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 16, 2022, 6:29am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/4 "2022-08-16T06:29:15Z")

</div>

`Ref` protects the result from being spread across multiple rows. Another way to do it is to wrap the output in `[]`:

```julia
julia> combine(groupby(df, :x), :y => (x -> [reduce(vcat, x)]) => :y)
2×2 DataFrame
 Row │ x y
     │ String Array…
─────┼──────────────────────
   1 │ a [1, 2, 3, 4]
   2 │ b [5, 6, 7, 8]

```

(in this case the vector is unwrapped, but its only element is another vector and unwrapping is not recursive)

---

<div class="post-metadata">

**Author:** ![ethomag](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ethomag/32/421_2.png) [@ethomag](https://discourse.julialang.org/u/ethomag)\
**Post date:** [August 16, 2022, 7:20am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/5 "2022-08-16T07:20:12Z")

</div>

Thanks @bkamins for your explanation, makes perfect sense!

I just encountered another issue with my de-fragmentation. I need each group in `gd` to be sorted according to `sampleno` (which indicates the start sample number in each packet), so that the resulting `samples` are properly ordered when concatenated.

I solved this by doing:

```julia
gd = groupby(df, [:frame, :subframe, :seq] )

for g in gd
    sort!(g, :sampleno)
end

```

Is there a better / simpler way ?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [August 16, 2022, 7:22am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/6 "2022-08-16T07:22:04Z")

</div>

Unless you need to preserve the order in the parent `df` you could just sort that?

---

<div class="post-metadata">

**Author:** ![ethomag](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ethomag/32/421_2.png) [@ethomag](https://discourse.julialang.org/u/ethomag)\
**Post date:** [August 16, 2022, 7:29am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/7 "2022-08-16T07:29:44Z")

</div>

Ah, ok so the order is guaranteed to be preserved. Great! Thanks.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [August 16, 2022, 7:31am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/8 "2022-08-16T07:31:28Z")

</div>

I think so, although @bkamins might correct me - I often get confused as to which operations are allowed to reorder things, but I think the within-group ordering of things should not get scrambled (otherwise things like calculating diffs, lags, or leads within group wouldn’t work).

---

<div class="post-metadata">

**Author:** ![ethomag](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ethomag/32/421_2.png) [@ethomag](https://discourse.julialang.org/u/ethomag)\
**Post date:** [August 16, 2022, 7:49am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/9 "2022-08-16T07:49:21Z")

</div>

At least it seems so:

```julia
df = # ...
gd = groupby(df, [:frame, :subframe, :seq] )
all(issorted(g, :sampleno) for g in gd)
# false

sort!(df, [:frame, :subframe, :seq, :sampleno])
gd = groupby(df, [:frame, :subframe, :seq] )
all(issorted(g, :sampleno) for g in gd)
# true

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 16, 2022, 7:20pm UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/10 "2022-08-16T19:20:44Z")

</div>

> [@nilshg](#):
>
> within-group ordering of things should not get scrambled

Yes - within group ordering is preserved always.

What is not guaranteed currently in DataFrames.jl with respect to row order are only two things:

- in `groupby` if you DO NOT pass `sort` kwarg then group order is undefined (i.e. DataFrames.jl picks the fastest algorithm it has available and can either sort groups or not sort them); use `sort=true` to sort groups and `sort=false` to preserve order of appearance
- in joins (except `leftjoin!`) row order is undefined - we will likely add an option to define row order in the future releases

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [August 18, 2022, 5:30pm UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/11 "2022-08-18T17:30:30Z")

</div>

You could get the same result, by reversing (in a sense) the order of operations.

```julia
combine(groupby(flatten(df,:y), :x), :y=>Ref=>:y)

```

perhaps put in this form is more understandable.

```julia
combine(groupby(flatten(df,:y), :x), :y=>(x->[x])=>:y)

```

or in a less usual form

```julia
combine(x->[flatten(x,:y).y], groupby(df, :x))

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [August 19, 2022, 5:57am UTC](https://discourse.julialang.org/t/dataframes-jl-vcat-column-data-from-different-rows/85809/12 "2022-08-19T05:57:31Z")

</div>

I wonder if it is possible with this last method

```julia
combine(f::Base.Callable, gd::GroupedDataFrame; args...)

```

to rename the column (s) involved, as is possible in the classic structure of mini-language.  
And if it is not available, what are the possibilities and / or contraindications for its implementation?