# Concatenate and then unconcatenate

**URL:** <https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538>\
**Category:** General Usage\
**Tags:** question\
**Created:** [February 7, 2019, 9:30am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538 "2019-02-07T09:30:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 7, 2019, 9:30am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/1 "2019-02-07T09:30:22Z")

</div>

Is there some nice solution for vertically concatenating arrays into one matrix, and then “unconcatenate” a similar matrix to equally sized arrays as we started with?

Something like this:

```julia
small_matrices = [rand(1,2), rand(3,2), rand(5, 2)]
large_matrix = vcat(small_matrices...)
similar_matrix = sin.(large_matrix)
small_matrices2 = magic(similar_matrix)
@assert small_matrices2 == map.(sin, small_matrices)

```

**Background:** I have a `Dict{String, Matrix{Float}}` (the dimensions of all the arrays are `n × 2`) of pixel coordinates of a tracked animal. I need to calibrate these coordinates to real world values. The calibration of the camera that recorded the trajectories of the animals was done a la calibration toolbox in matlab (waving a checkerboard in front of the camera).

Currently there is no equivalent auto-detection of the checkerboards in Julia, so I’m forced to use matlab. But because it’s matlab I kind of want to work with a single large matrix (and not multiple small matrices) for speed. Once matlab is done converting the coordinates from pixels to real-world, I want to redistribute the now calibrated coordinates back to the `Dict`. So I need to divide the large matrix I got back from matlab into the same shapes of small matrices I had before. The way I’m doing this now is by saving a vector of tuples with three indices: the key to the dictionary, the row index in the small matrix, the row index in the large matrix.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 7, 2019, 9:41am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/2 "2019-02-07T09:41:59Z")

</div>

Are `small_matrices` of the same size?

I have been experimenting with

> **[GitHub - tpapp/NestedViews.jl: Lazy combination and splitting of arrays.](https://github.com/tpapp/NestedViews.jl)**
>
> Lazy combination and splitting of arrays. Contribute to tpapp/NestedViews.jl development by creating an account on GitHub.

eg

```julia
using NestedViews # add from repo, not registered
combine(small_matrices)

```

Also, there is

> **[GitHub - JuliaData/SplitApplyCombine.jl: Split-apply-combine strategies for...](https://github.com/JuliaData/SplitApplyCombine.jl)**
>
> Split-apply-combine strategies for Julia. Contribute to JuliaData/SplitApplyCombine.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 7, 2019, 9:46am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/3 "2019-02-07T09:46:00Z")

</div>

> [@Tamas\_Papp](#):
>
> Are `small_matrices` of the same size?

No. They can have any number of rows (but always 2 columns).

> [@Tamas\_Papp](#):
>
> JuliaData/SplitApplyCombine.jl

Looks promising, I’ll try that out!

Thanks @Tamas_Papp!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 7, 2019, 9:59am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/4 "2019-02-07T09:59:01Z")

</div>

Hmmm, in a way, I think I’m looking for something similar to what matlab’s [`mat2cell`](https://www.mathworks.com/help/matlab/ref/mat2cell.html) does…

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 7, 2019, 2:53pm UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/5 "2019-02-07T14:53:16Z")

</div>

How about the below… It’s perhaps similar to the approach you’re currently using. I wouldn’t expect there to be a much easier solution, since the presence of a dictionary makes your problem quite specific.

```julia
function package(dict)
    rows = 1
    order = [(s = size(v,1); (k, s, (rows+=s)-s)) for (k,v) = dict]
    data = similar(first(values(d)), rows-1, 2)
    for (k,s,r) = order
        data[r:r+s-1,:] .= dict[k]
    end
    order,data
end

function update!(dict, order, data)
    for (k,s,r) = order
        dict[k] .= data[r:r+s-1,:]
    end
end

function convert!(data)
    data .= sin.(data)
end

dict = Dict("a" => rand(3,2), "b" => rand(1,2), "c" => rand(5, 2))
display(dict)

order,data = package(dict)

convert!(data)

update!(dict, order, data)
display(dict)

```

Output:

```julia
Dict{String,Array{Float64,2}} with 3 entries:
  "c" => [0.824289 0.957146; 0.732318 0.649147; … ; 0.974205 0.127819; 0.930694 0.259031]
  "b" => [0.571219 0.913474]
  "a" => [0.213684 0.511909; 0.295163 0.996942; 0.574854 0.688305]
Dict{String,Array{Float64,2}} with 3 entries:
  "c" => [0.734065 0.817551; 0.668595 0.604507; … ; 0.827255 0.127471; 0.802035 0.256144]
  "b" => [0.540658 0.791631]
  "a" => [0.212061 0.489842; 0.290896 0.839815; 0.543713 0.635229]

```

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 7, 2019, 2:59pm UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/6 "2019-02-07T14:59:06Z")

</div>

Yap, that’s more or less what I’m doing now. OK, I kind of hoped there would be a super-duper awesome way to do this. No worries, I got it working, but I can’t do it with out saving a collection of indices that help convert the large matrix to the small ones.

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [February 7, 2019, 4:23pm UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/7 "2019-02-07T16:23:41Z")

</div>

> [@yakir12](#):
>
> large\_matrix = vcat(small\_matrices…)

Instead of splatting, you can use a more optimized `reduce(vcat, small_matrices)`.

```julia
julia> small_matrices = [rand(1,2), rand(3,2), rand(5, 2)];

julia> @btime vcat($small_matrices...)
  115.738 ns (2 allocations: 256 bytes)
9×2 Array{Float64,2}:
 0.297374 0.448472
 0.746159 0.662536
 0.402667 0.0379419
 0.556125 0.0179574
 0.978084 0.0109701
 0.846586 0.347173
 0.336596 0.601298
 0.685396 0.159486
 0.409517 0.553826

julia> @btime reduce(vcat, $small_matrices)
  78.344 ns (1 allocation: 224 bytes)
9×2 Array{Float64,2}:
 0.297374 0.448472
 0.746159 0.662536
 0.402667 0.0379419
 0.556125 0.0179574
 0.978084 0.0109701
 0.846586 0.347173
 0.336596 0.601298
 0.685396 0.159486
 0.409517 0.553826

```

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 7, 2019, 10:00pm UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/8 "2019-02-07T22:00:39Z")

</div>

Or a regular for loop with a pre-allocated array for much better performance (at the cost of more code):

```julia
function fast_vcat(M)
    n = mapreduce(x -> size(x,1), +, M)
    V = similar(M[1], n, 2)
    r = 1
    for m = M
        s = size(m,1)
        @inbounds for j=1:2, i=1:s
            V[r+i-1,j] = m[i,j]
        end
        r += s
    end
    V
end

```

(This version is hard-coded to two columns, which gives a small performance boost.) Test:

```julia
julia> @btime vcat($M...);
  140.050 ns (2 allocations: 256 bytes)

julia> @btime reduce(vcat, $M);
  105.261 ns (1 allocation: 224 bytes)

julia> @btime fast_vcat($M);
  53.180 ns (1 allocation: 224 bytes)

julia> vcat(M...) == reduce(vcat, M) == fast_vcat(M)
true

```

However, I have a feeling that chasing nanoseconds is pointless in this case, since the data will be passed back and forth to MATLAB and processed.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 8, 2019, 7:21am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/9 "2019-02-08T07:21:06Z")

</div>

Very nice. Yea, while speed is always welcomed, I was hoping there would be some macro or package out there that does the unconcatenation. I can’t seem to see how [SplitApplyCombine.jl](https://github.com/JuliaData/SplitApplyCombine.jl) would do the trick because there is no way for the combine part to know to which indices the parts belong (the same kind of information `dim1Dist,...,dimNDist` convey in matlab’s `mat2cell`).

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 8, 2019, 9:09am UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/10 "2019-02-08T09:09:06Z")

</div>

I think CatViews.jl aims at solving this problem?

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 12, 2019, 8:01pm UTC](https://discourse.julialang.org/t/concatenate-and-then-unconcatenate/20538/11 "2019-02-12T20:01:27Z")

</div>

> [@baggepinnen](#):
>
> CatViews.jl

Works! Thanks!