# Concatenating in comprehension

**URL:** <https://discourse.julialang.org/t/concatenating-in-comprehension/48818>\
**Category:** General Usage\
**Created:** [October 22, 2020, 1:22pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818 "2020-10-22T13:22:19Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![HenriDeh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrideh/32/8316_2.png) [@HenriDeh](https://discourse.julialang.org/u/HenriDeh)\
**Post date:** [October 22, 2020, 1:22pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/1 "2020-10-22T13:22:20Z")

</div>

Hello,

```julia
[rand(10,10) rand(10,10)
 rand(10,10) rand(10,10)]

```

Makes a 20 x 20 Matrix. But

```julia
[rand(10,10) for i in 1:2, j in 1:2]

```

makes a `2×2 Array{Array{Float64,2},2}`.

I would like to know if there exists a comprehension syntax that would output the same as the first example. That way I can cat as many matrices as I want. Of course I use random matrices for the MWE but using rand(20,20) is not what I’m looking for.

Many thanks.

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [October 22, 2020, 1:43pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/2 "2020-10-22T13:43:13Z")

</div>

You can write `hvcat(ntuple(_->2, 2), (rand(10,10) for i in 1:2, j in 1:2)...)` but that’s not so pretty. `reduce(hcat, [rand(10,10) for i in 1:2, j in 1:2])` makes a 10×40 Matrix. I’d like `reduce(cat, [rand(10,10) for i in 1:2, j in 1:2])` to make a 10×10×2×2 Array, but it doesn’t yet.

---

<div class="post-metadata">

**Author:** ![yha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yha/32/3502_2.png) [@yha](https://discourse.julialang.org/u/yha)\
**Post date:** [October 22, 2020, 2:00pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/3 "2020-10-22T14:00:55Z")

</div>

You can do

```julia
reduce(vcat, reduce(hcat, rand(10,10) for i=1:2) for j=1:2)

```

This allocates intermediate results, which you may be able to avoid with lazy `hcat` from `LazyArrays.jl` if it matters.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 22, 2020, 5:51pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/4 "2020-10-22T17:51:54Z")

</div>

Since this is why I am adopting Julia, and we are never sure about how clear is that for people coming from other backgrounds (particularly I have that experience with students coming from python), I would like to point that you could also write your own function. For example, this would do the trick:

```julia
julia> function m(gen,n,m,N)
         M = Matrix{Float64}(undef,N*n,N*m)
         for i in 1:N
           kn = (i-1)*n+1
           for j in 1:N
             km = (j-1)*m+1
             M[kn:kn+n-1,km:km+m-1] .= gen(m,n)
           end
         end
         M
       end
m (generic function with 1 method)

```

where `gen` is the function that generates each sub-matrix, (in your mwe `rand`), `n` and `m` are the dimensions of that submatrix, and `N` is the total number of repetitions. For `N` relatively larger than 2 (I did not benchmark every case), this function is faster than the compact alternatives:

```julia
julia> @btime m(rand,10,10,100);
  4.568 ms (10002 allocations: 16.17 MiB)

julia> @btime reduce(vcat, reduce(hcat, rand(10,10) for i=1:100) for j=1:100);
  108.475 ms (28098 allocations: 779.97 MiB)

```

and, in addition, it can be easily parallelized:

```julia
julia> function m(gen,n,m,N)
         M = Matrix{Float64}(undef,N*n,N*m)
         Threads.@threads for i in 1:N
           kn = (i-1)*n+1
           for j in 1:N
             km = (j-1)*m+1
             M[kn:kn+n-1,km:km+m-1] .= gen(m,n)
           end
         end
         M
       end

julia> @btime m(rand,10,10,100);
  1.500 ms (10023 allocations: 16.18 MiB)

julia> Threads.nthreads()
4

```

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [October 22, 2020, 7:22pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/5 "2020-10-22T19:22:09Z")

</div>

Good point. You can also use `reshape` to avoid having to do index calculations:

```julia
julia> function m3(gen,n,m,N)
         M = Matrix{Float64}(undef,N*n,N*m)
         M4 = reshape(M, (n,N,m,N))
         for j in 1:N
           for i in 1:N
             M4[:,i,:,j] .= gen(n,m) # n,m reversed from m()
           end
         end
         M
       end
m3 (generic function with 1 method)

julia> m3(ones,2,3,4) == m(ones,2,3,4) # not a strong test!
true

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 23, 2020, 7:07am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/6 "2020-10-23T07:07:10Z")

</div>

Why do we need comprehension if the hvcat syntax is already so compact?

```julia
M1 = rand(3,5); M2 = rand(3,5); M3 = rand(2,5); M4 = rand(2,5);
MM = hvcat((2),M1,M2,M3,M4) # concatenate two in each block row

```

makes a `5×10 Array{Float64,2}:`

---

<div class="post-metadata">

**Author:** ![ctkelley](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ctkelley/32/10684_2.png) [@ctkelley](https://discourse.julialang.org/u/ctkelley)\
**Post date:** [October 23, 2020, 10:12am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/7 "2020-10-23T10:12:30Z")

</div>

This version is also easier for a novice, like a student coming from Matlab, Pythod, or C to understand. That’s a big deal in the education business.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [October 23, 2020, 10:24am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/8 "2020-10-23T10:24:55Z")

</div>

Using @rafael.guerra suggestion an answer to the original question can be

```julia
hvcat((2), [rand(5, 5) for _ in 1:4]...)

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 23, 2020, 10:51am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/9 "2020-10-23T10:51:26Z")

</div>

> [@Skoffer](#):
>
> `hvcat((2), [rand(5, 5) for _ in 1:4]...)`

And it is as fast and allocates the same thing as the manual version:

```julia
julia> @btime hvcat((100), [rand(10, 10) for _ in 1:10000]...)
  4.290 ms (10020 allocations: 16.79 MiB)

```

I guess this is the best answer (is it possible to parallelize this)?

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [October 23, 2020, 11:00am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/10 "2020-10-23T11:00:44Z")

</div>

> hvcat((2), [rand(5, 5) for \_ in 1:4]…)

Could you parse it and explain each term, please?

---

<div class="post-metadata">

**Author:** ![HenriDeh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrideh/32/8316_2.png) [@HenriDeh](https://discourse.julialang.org/u/HenriDeh)\
**Post date:** [October 23, 2020, 11:17am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/11 "2020-10-23T11:17:11Z")

</div>

Thank you all for your suggestions. I did not know hvcat but it seems to be the cleanest syntax while being efficient. Not that it mattered a lot but it’s good to know how to do things efficiently.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [October 23, 2020, 11:54am UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/12 "2020-10-23T11:54:30Z")

</div>

@lmiq  
I doubt that it can be parallelized, since it’s internal `Base` function. Yet on the other hand, I also doubt that it would be useful, since this function is mostly doing memory operations and memory operations rarely profit from parallelization (memory is much slower than CPU). As an example compare these two implementations (they are not very efficient, but it is not important, since we use the same operations in both versions)

```julia
using BenchmarkTools

x = [rand(10000) for _ in 1:100]
y = Vector{Float64}(undef, 1000000)

function f1!(y, x)
    for i in axes(x, 1)
        for j in axes(x[i], 1)
            y[(i - 1)*length(x[1]) + j] = x[i][j]
        end
    end

    return y
end

function f2!(y, x)
    Threads.@threads for i in axes(x, 1)
        for j in axes(x[i], 1)
            y[(i - 1)*length(x[1]) + j] = x[i][j]
        end
    end

    return y
end

```

and result is

```julia
julia> Threads.nthreads()
8

julia> @btime f1!($y, $x);
@btime f2!($y, $x);
  1.305 ms (0 allocations: 0 bytes)

julia> @btime f2!($y, $x);
  1.201 ms (41 allocations: 5.89 KiB)

```

The speedup that you observe was mainly related to the parallelization of the generation of the tables, which uses `rand` and it is a rather costly operation.

@Juan  
From `hvcat` documentation:

> Horizontal and vertical concatenation in one call. This function is called for block matrix  
> syntax. The first argument specifies the number of arguments to concatenate in each block  
> row.  
> If the first argument is a single integer n, then all block rows are assumed to have n block  
> columns.

So, the first argument to `hvcat` in our example is `(2)` and it means, that we only want two columns. Consider for example

```julia
julia> hvcat((2), 1, 2, 3, 4, 5, 6)
3×2 Array{Int64,2}:
 1 2
 3 4
 5 6

julia> hvcat((3), 1, 2, 3, 4, 5, 6)
2×3 Array{Int64,2}:
 1 2 3
 4 5 6

```

Now, `hvcat` do not accept vectors of matrices as an input, so I should have written

```julia
hvcat(2, rand(5, 5), rand(5, 5), rand(5, 5), rand(5, 5))

```

which is of course not scalable and inconvenient if we have a large number of matrices. But `hvcat` is so called [Vararg function](https://docs.julialang.org/en/v1/manual/functions/#Varargs-Functions) and I’ve used splatting syntax, when you add ellipsis to the last argument and julia split it into proper number of positional arguments. For easier understanding it can be written in two lines

```julia
rand_matrices = [rand(5, 5) for _ in 1:4]
hvcat(2, rand_matrices...) # which during parsing turns internally to hvcat(2, rand_matrices[1], rand_matrices[2], rand_matrices[3], rand_matrices[4])

```

So as a conclusion, I generated vector of 4 matrices 5x5 and said to `hvcat` to reorganize them in two block columns.

@HenriDeh  
All kudos should go to @rafael.guerra 🙂

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 23, 2020, 1:12pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/13 "2020-10-23T13:12:02Z")

</div>

> [@Skoffer](#):
>
> I also doubt that it would be useful

Actually in this case it is generating the sub-matrices of dimensions `(m,n)` multiple times, and that can be parallelized (this is what happens in the custom function I provided before, which parallelizes very well, because it is embarrassingly parallel).

In your example you moved the generation of the random matrices out of the benchmark, so that does not correspond exactly to the `hvcat` solution.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 23, 2020, 1:25pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/14 "2020-10-23T13:25:47Z")

</div>

More variations on the same theme: is the following a proper way of collecting all the matrices Mi before hvcat?

```julia
M1 = rand(3,5); M2 = rand(3,5); M3 = rand(2,5); M4 = rand(2,5);
mlist = [M1,M2,M3,M4]
MM = hvcat((2), mlist...) # concatenates two in each block row

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 23, 2020, 1:36pm UTC](https://discourse.julialang.org/t/concatenating-in-comprehension/48818/15 "2020-10-23T13:36:29Z")

</div>

It is, _because_`mlist` is only addressing the positions, i. e.:

```julia
julia> x = rand(2) ; y = rand(2);

julia> m = [x, y]
2-element Array{Array{Float64,1},1}:
 [0.2383336264485605, 0.8321815241645099]
 [0.5923411990097951, 0.9487571368376246]

julia> m[1]
2-element Array{Float64,1}:
 0.2383336264485605
 0.8321815241645099

julia> m[1][1] = 0.
0.0

julia> x
2-element Array{Float64,1}:
 0.0
 0.8321815241645099

```