Slicing sparse and dense matrices and allocations

Hi, I am optimizing some old non-performance critical code of mine and run into a lot of allocations when using slicing operations with sparse and dense matrices.

I distilled it to this MWE

# -----------------------------------
# Initialize everything. This comes from somewhere else in the actual code.
using SparseArrays
p = 6; n = 14; Tf = 248;Tfn = n*Tf;
AzeroAp = ones(n,n*(p+1));      # collects [A0 -A1 -A2 ... -Ap]
H_Int = zeros(Int,Tfn,Tfn)              # initializes the H matrix to get the linear indices

# fill the block-diagonals and off-diagonals with p+1 (nxn) matrices
for ij = 0:p
    for ii = 1:div(Tfn,n)-ij
        H_Int[ ij*n + (ii-1)*n + 1 : n + (ii-1)*n +  ij*n, (ii-1)*n + 1 : n + (ii-1)*n] = ones(Int,n,n)
    end
end
H_sp = sparse(H_Int)               # set everything else to 0 (density around 2%, sparsity 98%)
AzeroAp_LinInd = LinearIndices(AzeroAp) # get the linear indices of  [A0 -A1 -A2 ... -Ap]

# ----------------------------------
# THIS IS THE CRITICAL PART
@time for ij = 0:p               # loop over the block-diagonal and off diagonals
    for ii = 1:div(Tfn,n)-ij
        H_sp[ ij*n + (ii-1)*n + 1 : n + (ii-1)*n +  ij*n, (ii-1)*n + 1 : n + (ii-1)*n] = AzeroAp_LinInd[ :, 1 + (ij-0)*n : n + (ij-0)*n]
    end
end

What the code does

I am creating the the following big block-banded matrix[1]:

\begin{equation*} \mathbf{H}=\left[\begin{array}{cccccccc} \mathbf{A_0} & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & \cdots & \cdots & \cdots & \cdots & \cdots & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} \\ -\mathbf{A}_1 & \mathbf{A_0} & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & \cdots & \cdots & \cdots & \cdots & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} \\ -\mathbf{A}_2 & -\mathbf{A}_1 & \mathbf{A_0} & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & \cdots & & & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} \\ \vdots & \ddots & \ddots & \ddots & \ddots & \ddots & & \vdots \\ -\mathbf{A}_p & \cdots & & -\mathbf{A}_1 & \mathbf{A_0} & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & & \vdots \\ \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & & & & \ddots & \ddots & \ddots & \vdots \\ \vdots & & \ddots & & \ddots & \ddots & \ddots & \vdots \\ \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & \cdots & \mathbf{0}_{\boldsymbol{n} \times \boldsymbol{n}} & -\mathbf{A}_p & \cdots & -\mathbf{A}_2 & -\mathbf{A}_1 &\mathbf{A_0} \end{array}\right] \end{equation*}

It has \mathbf{A}_0 on the main block diagonal, \mathbf{A}_1 on the diagonal below that and so on until \mathbf{A}_p. p is always a small number, max is 6, n is also less than 20, while we have Tf block-rows, where Tf could be 200, so total Tf*n \times Tf*n.

In the code above, the variable AzeroAp := \left[ \mathbf{A}_0, -\mathbf{A}_1, \dots, -\mathbf{A}_p \right].

The objective of the code is to map the linear indices of AzeroAp to the corresponding linear indices of these matrices in \mathbf{H}.[2]. For a small example with p=1 and n=2

\mathbf{H_{Int}}=\begin{bmatrix} 1&5&9&13\\ 2&6&10&14\\ 3&7&11&15\\ 4&8&12&16\\ \end{bmatrix}

The indices corresponding to \mathbf{A_0} are

\begin{bmatrix} 1&5\\ 2&6\\ \end{bmatrix}, \text{and}\begin{bmatrix} 11&15\\ 12&16\\ \end{bmatrix}

or in vector form [1, 2, 5, 6, 11, 12, 15,16]', while the indices corresponding to \mathbf{A_1} are [3, 4, 7, 8]' in vector form. So if I call H[3,4,7,8] = AzeroAp[5, 6, 7, 8] I will update the Hmatrix. The last loop essentially gives me these linear indices for that updating.

The question

When we come to the section with the crucial part, where I map the indices, this allocates like hell. For the above example it uses 8GB of memory.

julia> @time for ij = 0:p
           for ii = 1:div(Tfn,n)-ij
               H_sp[ ij*n + (ii-1)*n + 1 : n + (ii-1)*n +  ij*n, (ii-1)*n + 1 : n + (ii-1)*n] = AzeroAp_LinInd[ :, 1 + (ij-0)*n : n + (ij-0)*n]
           end
       end
  6.088194 seconds (51.16 k allocations: 8.645 GiB, 38.78% gc time)

Before coming here, as it is normal nowadays, I asked AI and the explanation was that slicing sparse and dense matrices allocates and that I should use setindex [3]. Then, the AI suggested this “fix”

for ij = 0:p
    for ii = 1:div(Tfn, n) - ij
        row_start = ij * n + (ii - 1) * n + 1
        row_end = n + (ii - 1) * n + ij * n
        col_start = (ii - 1) * n + 1
        col_end = n + (ii - 1) * n

        # Directly update the sparse matrix
        for i = row_start:row_end
            for j = col_start:col_end
                H_sp[i, j] = AzeroAp_LinInd[i - row_start + 1, j - col_start + 1]
            end
        end
    end
end

It effing works and I don’t understand why. What am I doing wrong? Why is it that when it is split across rows and columns works?

julia>  0.108702 seconds (1.35 M allocations: 36.757 MiB)

TLDR: why does the last for loop in the MWE allocate a lot, I thought I was just looking into the matrices and mappign their values from the AzeroAp to H_sp


Additional info for the curious

reveal

While this is unncessary for the question at hand you might be wondering what the hell am I doing.
The main problem is that in every iteration (say from 20000), I get new \mathbf{A} matrices, so I need to update the \mathbf{H} matrix with the new [\mathbf{A_0} \dots \mathbf{A_p}]. So I thought its easiest to save the linear indices, so that I can do

H_sp[Lind] = AzeroAp[Rind]

So before the main loop I get the linear indices vectors, Lind and Rind, respectively, once with the above code. In that sense it is not performance critical, because it is only ran once but still I thought I am doing something wrong with that many allocations. Other than that the code and the updating work very well.

Maybe there is a much easier way generate Lindand Rind but I didn’t know how and that is how I came up with the above solution. I also tried all the block-banded packages, as well as the .nzval property of sparse matrices but none worked in the way I wanted them to.


  1. Yes, I am aware and did all try the BlockBanded and sparse BlockBanded packages in the Julia ecosystem ↩︎

  2. I will discuss this design choice later as to not make the exposition too large, but there are definitely other ways to achieve the same result. ↩︎

  3. honestly this is what I thought I was doing ↩︎

EDIT: misdiagonosed the issue because this has to do with SparseArrays. See below comments. That said, I would still make the recommendation of finding/making a custom array type based on dense blocks in a Toeplitz structure.

The issue is that slicing in Julia (except as the target of an assignment) allocates a copy of the region. This can be avoided by using the view function (instead of getindex, which is used by the A[i, j] syntax) or using the @view or @views macro to make this substitution for you.

Replacing

H_sp[#= ... =#] = AzeroAp_LinInd[ :, 1 + (ij-0)*n : n + (ij-0)*n]
# with
H_sp[#= ... =#] = @view AzeroAp_LinInd[ :, 1 + (ij-0)*n : n + (ij-0)*n]

should remove those allocations.


Now a couple of thoughts:

This “block-banded” matrix is also sometimes called a block Toeplitz matrix. Unfortunuately, it looks like ToeplitzMatrices.jl only handles non-block Toeplitz matrices. If you make the individual blocks using StaticArrays.jl it might work but I’m guessing not.

Regardless, you might benefit from implementing a type for this structure yourself (maybe BlockArrays.jl can help a little? maybe not). A Toeplitz matrix can be efficiently multiplied (or inverted, among a few other operations) by embedding it in a circulant matrix and using the fast Fourier transform, but that’s a touch more complicated in the block case unless the blocks are also Toeplitz.

No, that’s not the culprit here. Using @view, putting everything in a function, and using @btime still yields 8GB of allocations.

The basic problem is that setindex! for sparse matrices is very hard to implement efficiently, especially if there is a possibility that you may be inserting new nonzero entries. If you want to mutate existing nonzero values in a sparse matrix it’s vastly more efficient to do so on the column-sorted nonzeros(matrix) array instead if you can, although this requires more care with indexing.

In particular, the sparse-matrix setindex! with a vector of indices allocates because it assumes you might be inserting new non nonzeros: SparseArrays.jl/src/sparsematrix.jl at 7abbefaedb456d0b24a94873c01a345780fe07f1 · JuliaSparse/SparseArrays.jl · GitHub

It could be made much more efficient if setindex! knew that the indices are sorted and no new nonzeros would be inserted. (Not sure what the API would look like?)

Thanks to you both.

I also thought that viewing would help and tried that, but it didn’t help and I didn’t include it in my post.

I did try block arrays, and the block banded suites and ended up on this solution as the best.

I should be honest and say that my linear algebra is not up to par, I should finally read about toeplitz structures. Few years ago I was coding this in Matlab and asked in MatlabCentral what is the best way to do this and a person gave me an amazing almost oneliner based on a toeplitz structure that I use to this day.

Matlab code

Link to the discussion on Matlab central

function [L,R]=makeIndices(n,p,T)
  
Icell=mat2cell(reshape(1:n^2*p,n,[]), n,n*ones(1,p) );
Icell=[zeros(n), Icell, zeros(n)];
idx=toeplitz(min(p+2,1:T), [1,repelem(p+2,T-1)]  );
 Q = cell2mat(Icell(idx));
 L=logical(Q);
 R=nonzeros(Q);
end

I didn’t manage to translate this to Julia so I wrote the above implementation.

What I am invested in the above is, why does the AI code solve the problem? How does it fix the issue with the setindex? Is it to do that it goes columnwise? Also, I have to say that I am not touching any non-zeros.

Thanks to you both.

I also thought that viewing would help and tried that, but it didn’t help and I didn’t include it in my post.

I did try block arrays, and the block banded suites and ended up on this solution as the best.

I should be honest and say that my linear algebra is not up to par, I should finally read about toeplitz structures. Few years ago I was coding this in Matlab and asked in MatlabCentral what is the best way to do this and a person gave me an amazing almost oneliner based on a toeplitz structure that I use to this day.

Matlab code

Link to the discussion on Matlab central

function [L,R]=makeIndices(n,p,T)
  
Icell=mat2cell(reshape(1:n^2*p,n,[]), n,n*ones(1,p) );
Icell=[zeros(n), Icell, zeros(n)];
idx=toeplitz(min(p+2,1:T), [1,repelem(p+2,T-1)]  );
 Q = cell2mat(Icell(idx));
 L=logical(Q);
 R=nonzeros(Q);
end

I didn’t manage to translate this to Julia so I wrote the above implementation.

What I am interested in the above is, why does the AI code solve the problem? How does it fix the issue with the setindex? Is it to do that it goes columnwise? Also, I have to say that I am not touching any non-zeros.

Edit:

It could be made much more efficient if setindex! knew that the indices are sorted and no new nonzeros would be inserted. (Not sure what the API would look like?)

I found it extremely fast to access the indices with the .nzval property, but couldn’t figure out how to map that to my much much smaller matrix and the fact that the more I get near the bottom row, the more I lose blocks from my AzeroAp matrix. In the last block-row I only have \mathbf{A_0} and not the whole column-block AzeroAp'. Maybe if it were possible for the sparse matrix to give me the indices of the values behind nzval by dropping the zeros? I don’t know if this makes sense or if I am conveying my idea properly.

I think the allocation difference between your code the AI code is that the latter doesn’t actually use slicing. Instead of H_sp[ ij*n + (ii-1)*n + 1 : n + (ii-1)*n + ij*n, (ii-1)*n + 1 : n + (ii-1)*n] which creates a copy of this subset, the AI updates the matrix one element at a time with H_sp[i, j] which does not create a new copy.

Hm, somehow I didn’t get that, I thought it was a slice but you are correct.

Depending on what your doing with them, creating you own datatype may be very helpful. I threw this together with the help of an LLM:

At the very least it makes updating the blocks a lot easier.