How to stack two "cell arrays" together

Leon6 · April 16, 2023, 9:22pm

I have two cell arrays, one of them
A is a 27x1 cell, size(A{i}) is 360x180.
B is a 35x1 cell, and size(B{i}) is the same as the above, i.e., 360x180.

How do I stack them together to create a C that has a size of 360x180x62?

Thanks.

sylvaticus · April 16, 2023, 9:32pm

Do you mean that you have A, B that are vectors (i e. 1-dim arrays) of matrices (i e. 2 dims arrays) and you want to stack them in a 3 dims Array?

Leon6 · April 16, 2023, 9:35pm

I’m not familiar with all these terms, but basically, A has 27 matrices. In Matlab I can call them like this:
A{1}
A{2}
…
A{27}.

What I want to get is a C that stacks up all these 27 matrices as well as the 35 matrices from B, thus producing a matrix with the size of 360x180x62.

Thanks.

sylvaticus · April 16, 2023, 9:42pm

If all you want is to concatenate A and B (both vectors where each entry is a matrix) to a C vector containing all the matrices you can use C = vcat(A,B), but this will NOT produce a 3 dimensional array, but a nested vector of matrices.

rocco_sprmnt21 · April 16, 2023, 10:25pm

maybe this is what you are looking for

A=rand(360,180,27);
B=rand(360,180,35);

c=[A;;;B]
#or

c=hvncat(3,A,B)

DNF · April 16, 2023, 10:41pm

Is this a Matlab or Julia question?

In Julia, there is no such thing as a cell array, though Array{Any, N} is similar. But assuming you have two vectors of matrices, you can do

C = cat(A..., B...; dims=3)

mkitti · April 17, 2023, 1:08pm

This looks like a MATLAB question.

In MATLAB, see cat.

You might vwant something like cat(3, A{:}, B{:}).

For MATLAB questions in the future see
https://www.mathworks.com/matlabcentral/

This is not a MATLAB help forum.

Leon6 · April 17, 2023, 9:43pm

To clarify, this is a Julia question 100%. My apologies for the confusion. I just used the Matlab “cell array” concept to describe the array, as I do not know how it is called within Julia.

Leon6 · April 17, 2023, 9:49pm

Thanks for the recommended solution. Unfortunately, it does not work for me. My Julia would freeze every single time I tried this.

Leon6 · April 17, 2023, 9:49pm

Thanks for the suggestions. However, my A and B are vectors of matrices, instead of 3d arrays.

mkitti · April 17, 2023, 9:52pm

What is your A and B in Julia?

In Julia, I would do this.

julia> begin
           A = [zeros(UInt8, 360, 180) for i in 1:27]
           B = [zeros(UInt8, 360, 180) for i in 1:35]
       end;

julia> C = [A...;;; B... ];

julia> size(C)
(360, 180, 62)

You can access inidivudal matrices as follows.

julia> A[1];

julia> A[27];

julia> B[27];

julia> C[60];

Basically we do not need the concept of cell array in Julia. It’s just an array.

The literal equivalent of a cell array in Julia would be Array{N,Any} where N Array{N, Any} where N where each position of the array here could be anything.

julia> v = Vector{Any}(undef, 5);

julia> v[1] = 5
5

julia> v[2] = zeros(3,5)
3×5 Matrix{Float64}:
 0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0

julia> v[3] = true
true

julia> v[4] = 9.0
9.0

julia> v[5] = π
π = 3.1415926535897...

julia> v
5-element Vector{Any}:
    5
     [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]
 true
    9.0
    π = 3.1415926535897...

In your case, however, you know all the elements are going to be of type Matrix{Float64} so we can be more specific.

julia> typeof(A)
Vector{Matrix{UInt8}} (alias for Array{Array{UInt8, 2}, 1})

DNF · April 17, 2023, 10:00pm

I’m glad you found a solution, but I’d like to point out that it’s a solution to a very different problem than the one described in the Original Post. It shows how to concatenate 3d arrays, but you were asking about vectors of matrices (‘cell arrays of matrices’).

My proposed solution concerned itself vectors of matrices. If you feed it 3d arrays, it’s no surprise Julia froze, since it would try to compile a function with 4 million input arguments (360 * 180 * 62).

Leon6 · April 17, 2023, 10:47pm

You are exactly right. I got myself confused. As you mentioned, my A and B are not 3d arrays like the below.

A=rand(360,180,27);
B=rand(360,180,35);

Instead they are vectors of matrices like these:

begin
     A = [zeros(UInt8, 360, 180) for i in 1:27]
     B = [zeros(UInt8, 360, 180) for i in 1:35]
end;

As a result, your recommended solution as below works the best:

C = cat(A..., B...; dims=3)

Sorry for the confusion, but I’m glad this post now has solutions for both 3d arrays and vectors of arrays.

Mason · April 21, 2023, 5:25pm

Please don’t recommend to beginners that they splat arrays like this. You’re just going to cause them problems and pain if they use an array that’s too big since it’ll always incur a dynamic dispatch, and it creates large Tuples which can cause the compiler to freak out.

@Leon6 a more scalable way of doing this is with the stack function:

D = stack(vcat(A, B));

Here’s a performance comparison:

julia> begin
            A = [zeros(UInt8, 360, 180) for i in 1:27]
            B = [zeros(UInt8, 360, 180) for i in 1:35]
       end;

julia> @btime C = cat($A..., $B...; dims=3);
  2.071 ms (573 allocations: 3.88 MiB)

julia> @btime D = stack(vcat($A, $B));
  104.551 μs (3 allocations: 3.83 MiB)

Easy 20x improvement.

DNF · April 21, 2023, 5:52pm

I’d like to point out that stack isn’t available in Julia yet, so think there should be a big disclaimer on that.

Also, I have frequently found that splatting performs annoyingly well, for some reason, even for large arrays. The performance characteristics of splatting remains a bit of a mystery to me.

mkitti · April 21, 2023, 6:20pm

Since stack is unreleased as of Julia 1.8, you need to tell him how to do with pre-Julia 1.9.

jar1 · April 21, 2023, 6:23pm

stack is in Compat.jl.

Mason · April 21, 2023, 8:20pm

Yes, sorry I misremembered and thought it was in 1.8. The ammended advice is to do (once)

using Pkg
Pkg.add("Compat")

and then once that’s installed, you can do

using Compat: stack
D = stack(vcat(A, B));

rocco_sprmnt21 · April 22, 2023, 3:18pm

begin
    A = [rand(360, 180) for _ in 1:27]
    B = [rand(360, 180) for _ in 1:35]
end;

using BenchmarkTools

julia> @btime C=hvncat(3, A...,B...);
  543.363 ms (20024381 allocations: 397.52 MiB)

julia> @btime reshape(mapreduce(vec,vcat,A), (size(A[1])...,:));     
  41.736 ms (110 allocations: 186.39 MiB)
julia> @btime C = cat($A..., $B...; dims=3);
  6.810 ms (573 allocations: 30.70 MiB)

# cat seems 100X faster than hvncat(!?!?)

using Compat: stack

julia> @btime stack(vcat(A, B));
  6.774 ms (3 allocations: 30.65 MiB)

# using copyto! seems faster than stack

julia> @btime  begin
       C=similar(Array{Float64,3},(size(A[1])...,62));
       foreach(k->copyto!($C,1+360*180*(k-1), [A;B][k], 1, 360*180), 1:62)
       end
  4.009 ms (127 allocations: 30.69 MiB)

julia> @btime foreach(k->copyto!($C,1+360*180*(k-1), [A;B][k], 1, 360*180), 1:62)
  3.905 ms (123 allocations: 33.89 KiB)

julia> @btime begin
           C=similar(Array{Float64,3},(size(A[1])...,62));
           foreach(k->copyto!($C,1+360*180*(k-1), A[k], 1, 360*180), 1:27)
           foreach(k->copyto!($C,1+360*180*(k-1), B[k-27], 1, 360*180), 28:62)
       end
  4.167 ms (65 allocations: 30.65 MiB)

PS
I couldn’t use the copyto!() method which makes use of cartesianindices

sylvaticus · April 22, 2023, 3:29pm

Note that your second to last example doesn’t include the allocation of C…