How can I do list comprehension for vcat?

General Usage
1

I have many 1-d lists that I want to concatenate to have a 1-d list, probably in a list comprehension style. Following is how I can achieve using explicit for-loop

f = []
a = [1,2,3,4,6]
b = [3,4,5,20]
c = [8,4,5,20]
d = [8,1,90,40]
x=[a,b,c,d,a]
for a in x
    f= [f;a]
end
f  # I get 22-element Vector which is what I am looking for.

Thank you for your help in advance.

2 
julia> f = vcat(a,b,c,d,a)
22-element Vector{Int64}:
  1
  2
  3
  4
  6
  3
  4
  5
 20
  ⋮
  1
 90
 40
  1
  2
  3
  4
  6
3 Likes
3

or, use reduce(vcat, ...) if you want to work on the vector-of-vectors x:

a = [1,2,3,4,6]
b = [3,4,5,20]
c = [8,4,5,20]
d = [8,1,90,40]
x = [a,b,c,d,a]

julia> f = reduce(vcat, x)
22-element Vector{Int64}:
  1
  2
  3
  ⋮
  3
  4
  6
6 Likes
4

if you don’t necessarily have to use vcat

append!(f, x...)

instead if you want to use vcat :grinning:

vcat(x...)

[a;b;c;d;a]
1 Like
5

Splatting can have quite unpredictable performance. In general, reduce(vcat, x)

5 Likes
6

How could we write our own Julia loop competitively?
An attempt:

function vcat2(x::AbstractVector{<:AbstractVector})
    y = Array{eltype(x[begin])}(undef,sum(length,x))
    i = 1
    for u in x
        l = length(u)
        y[i:(i+l-1)] .= u
        i += l
    end
    return y
end

# Ex.1:
a, b, c, d = [1,2,3,4,6], [3,4,5,20], [8,4,5,20], [8,1,90,40]
x = [a,b,c,d,a]
@btime reduce(vcat, $x)                     #  80 ns (2 allocs: 336 bytes)
@btime vcat($x...)                          #  92 ns (1 alloc:  240 bytes)
@btime vcat2($x)                            #  85 ns (2 allocs: 336 bytes)

# Ex.2:
using Random
Random.seed!(123)
x = [rand(100) for _ in 1:100]
@btime reduce(vcat, $x)                     # 5.620 μs (3 allocs: 79 KiB)
@btime vcat($x...)                          # 5.740 μs (2 allocs: 78 KiB)
@btime vcat2($x)                            # 5.560 μs (2 allocs: 78 KiB)
1 Like
7 
julia> @btime vcat(x...);
  79.442 ns (1 allocation: 240 bytes)

julia> @btime reduce(vcat, x)  ; 
  83.731 ns (2 allocations: 336 bytes)

julia> @btime vcat2(x) ;                    
  72.587 ns (2 allocations: 336 bytes)
1 Like
8

Thank you very much for the lovely answer. Out of all the good answers, I used

reduce(vcat, x)

and it worked great until I had empty x. Empty x gave me the following error

ArgumentError: reducing over an empty collection is not allowed

Is there any way to resolve this issue? Thanks for the response.

9

You need to specify an init

julia> reduce(vcat, Int[])
ERROR: ArgumentError: reducing over an empty collection is not allowed

julia> reduce(vcat, Int[], init=Int[])
Int64[]
4 Likes
10

vcat with splatting does not suffer from this problem

a=rand(1:10, rand(1:10))
b=rand(1:10, rand(1:10))
c=rand(1:10, rand(1:10))
d=rand(1:10, rand(1:10))
e=rand(1:10, rand(1:10))
f=[]



x=[a,b,c,d,f,e]


vcat(x...)

y=[f,a,b,c,d,e,f]

Base.splat(vcat)(y)

even in terms of performance it does well

v=[rand(1:10, rand(0:10)) for _ in 1:100]

using BenchmarkTools
julia> @btime vcat(v...);
  873.684 ns (1 allocation: 4.06 KiB)

julia> @btime reduce(vcat, v, init=Int[]);
  19.600 μs (102 allocations: 206.09 KiB)

julia> vcat(v...)==reduce(vcat, v, init=Int[])
true

if you notice the number of allocations of Base.splat(vcat) is always 1 up to vectors with 256 elements. While that of reduce(vcat) is equal to the size of the vector v + 2


julia> v=[rand(1:10, rand(0:10)) for _ in 1:256]
256-element Vector{Vector{Int64}}:
 [8, 4, 8, 1, 1, 9]
 [4, 8, 6, 8, 10, 8, 6, 9, 4]
 [2, 3, 10, 5, 2]
 [6]
 [2, 3]
 [5, 4]
...
 [8, 4, 4, 10, 9, 1]
 [5]
 [8, 9]
 [3, 10, 9, 3, 3, 6, 9, 1, 9]
 []

julia> @btime vcat(v...);
  2.044 μs (1 allocation: 9.94 KiB)

julia> @btime reduce(vcat, v, init=Int[]);
  120.400 μs (258 allocations: 1.29 MiB)
1 Like
11

For some reason, the init slows it down quite a bit:

julia> @btime reduce(vcat, $v, init=Int[]);
  23.151 μs (101 allocations: 210.23 KiB)

julia> @btime reduce(vcat, $v);
  962.444 ns (2 allocations: 5.12 KiB)

There is a related open issue for mapreduce (#31137) — it seems like we didn’t optimize a sufficiently low-level method of reduce.

4 Likes