Efficient reshaping of 2D array by pairs of columns

General Usage
1

Hi,

I’m building up an MC code that has to run fast and efficiently. At one step I need to reshape a large array with two columns into an array with much less columns and many more rows. A very simple example woudl be that I want this

9×2 Array{Int64,2}:
 1  2
 3  4
 5  6
 7  8
 9 10
11 12
13 14
15 16
17 18

to become like this

3×6 Array{Int64,2}:
 1  2   7   8  13  14
 3  4   9  10  15  16
 5  6  11  12  17  18

Is there any simple and efficient way to do this kind of reshaping? I mean, with no intermediate array creation, and such that it is fast?

Thanks a lot,

Ferran.

2

Check out the reshape function.

3

Thanks, yes. I was actually asking about how to reshape not along one axis (=one column), but by pairs of columns. Notice in my actual problem I have an array of (NpNw) x 2 elements that I want to reshape into Np x (2Nw) form…

4

It’s unlikely you’ll find a way to do that without modifying the data in memory, since the Array type stores 2d arrays in column-major order. The first array will have 1 3 5 7 9 11 … in contiguous memory, the second will have 1 3 5 2 4 6 … in contiguous memory. To go from on to the other you’ll have to modify the contents of memory. You might be able to do this with a sequence of reshapes and transposes.

1 Like
5

You could probably eliminate the inner permutedims and maybe even the initial reshape by accumulating into a (2, 3, N) array to start with, but to get the final result you do have to move values around in memory.

julia> aa = [1  2
        3  4
        5  6
        7  8
        9 10
       11 12
       13 14
       15 16
       17 18]

julia> reshape(permutedims(reshape(permutedims(aa), (2, 3, 3)), (2,1,3)), (3, 6))
3×6 Array{Int64,2}:
 1  2   7   8  13  14
 3  4   9  10  15  16
 5  6  11  12  17  18
1 Like
6

Here’s one way to write the reshapes & permutations:

julia> using TensorCast

julia> @cast B[a,(c,b)] := A[(a,b),c]  a:3
3×6 reshape(PermutedDimsArray(::Array{Int64, 3}, (1, 3, 2)), 3, 6) with eltype Int64:
 1  2   7   8  13  14
 3  4   9  10  15  16
 5  6  11  12  17  18

Edit – This expands out to the following, with one permutation:

sz_a, sz_c = 3, size(A, 2)
tmp = PermutedDimsArray(reshape(A, (sz_a, :, sz_c)), (1, 3, 2))
B = reshape(tmp, (sz_a, :))
4 Likes
7

@contradict, how would you write your solution in terms of the Np and Nw variables defined by Ferran Mazzanti? Thank you.

8

Oh, thank you guys for the solutions. They seem to be fairly involved, so I’ll have to study them carefully…
Best,
Ferran.

9

Maybe like this?

rearrange(aa, Np, Nw) = reshape(permutedims(reshape(permutedims(aa), (2, Np, Nw)), (2,1,3)), (Np, 2*Nw))

First, get the pair alone. Then swap the two dimensions that need to be moved in memory. Then collapse out the added dimension.

In real code, it would probably be worth ensuring the function produces a sensible error when aa has an unexpected shape.

Although, now that I know about TensorCast, I’d probably use that!

1 Like
10

@contradict, thank your for the explanation, learning a lot about this. @mcabbott’s solution is turbo and can be written from his post edit as:

reshape(PermutedDimsArray(reshape(A, (Np, :, 2)), (1, 3, 2)), (Np, :))

It does not require Nw input …

1 Like