How to construct a large sparse matrix?

General Usage
, ,
1

Currently, I’m using

n = 10^5
M = sparse(BitArray(n,n))

But when n > 10^6 I run out of memory so this doesn’t work. I’m unsure what to do as the bellow syntax chucks an error

SparseMatrixCSC{Bool}(n,n)
MethodError: no method matching SparseMatrixCSC{Bool,Ti} where Ti<:Integer(::Int64, ::Int64)
2

Because BitArray(n,n) allocates the 10^6 x 10^6 array and then sparse transform it to a (full) sparse matrix

3 
help?> sparse                                                                                                                                                            
...
  sparse(I, J, V,[ m, n, combine])

  Create a sparse matrix S of dimensions m x n such that S[I[k], J[k]] = V[k]. The combine function is used to combine duplicates. If m and n are not specified, they    
  are set to maximum(I) and maximum(J) respectively. If the combine function is not supplied, combine defaults to + unless the elements of V are Booleans in which case  
  combine defaults to |. All elements of I must satisfy 1 <= I[k] <= m, and all elements of J must satisfy 1 <= J[k] <= n. Numerical zeros in (I, J, V) are retained as  
  structural nonzeros; to drop numerical zeros, use dropzeros!.

  For additional documentation and an expert driver, see Base.SparseArrays.sparse!.                                                                                      
                                                                                                                                                                         
     Example                                                                                                                                                             
    ≡≡≡≡≡≡≡≡≡                                                                                                                                                            

  julia> Is = [1; 2; 3];                                                                                                                                                 
                                                                                                                                                                         
  julia> Js = [1; 2; 3];                                                                                                                                                 
                                                                                                                                                                         
  julia> Vs = [1; 2; 3];                                                                                                                                                 
                                                                                                                                                                         
  julia> sparse(Is, Js, Vs)                                                                                                                                              
  3×3 SparseMatrixCSC{Int64,Int64} with 3 stored entries:                                                                                                                
    [1, 1]  =  1                                                                                                                                                         
    [2, 2]  =  2                                                                                                                                                         
    [3, 3]  =  3

Also, sprand might be of use.

1 Like
4

Thanks, I was looking for

n = 10^6
sparse([],[],Bool[],n,n)
5

Note that inserting elements on a non stored zero in a sparse matrix is quite expensive. If possible it might be worth building the matrix from I, J, V and then convert it to a sparse matrix to do operations that does not change the sparsity pattern.

7 Likes
6

In some applications you can use other construction methods too, e.g. spdiagm for sparse matrices with a banded structure, or Kronecker products (kron) of smaller sparse matrices for sparse matrices arising from regular grids (such as in finite-difference schemes).

2 Likes
7

thank you for these extra tips, really helpful!