Saving sparse matrix

General Usage
1

How to save a sparse matrix in Julia? I want the saved file to be small since my Matrix size is about (2^26,2^26).
Thanks.

2

I would just save nonzero indexes and values as CSV:

using SparseArrays, DataFrames, CSV
M = sprand(10, 10, 0.2)
I, J, V = findnz(M)
df = DataFrame([:I => I, :J => J, :V => V])
CSV.write("/tmp/spmatrix.csv", df)
5 Likes
3

Thank you, the problem is that CSV file is very large, for example in python I use *.npz which is 2-5 MB but here it can go up to 500 MB.

4

I don’t know this format, but you can also compress the CSV, eg with gzip.

Alternatively, you can save a bit of space by using the CSC representation (ie export directly from the fields), but I don’t think it is worth it.

1 Like
5

Great answer. Minor remark in passing: with LinearAlgebra.I, I is no longer a great choice in I, J, V. :disappointed_relieved: R, C, V or M, N, V are viable alternatives I’ve seen used.

6

For production code I would wrap it in a function so this should not matter.

1 Like
7

Very true!

using SparseArrays, DataFrames, CSV, LinearAlgebra
function save_sparse_matrix(M)
    I, J, V = findnz(M)
    println("I = ", I)
    df = DataFrame([:I => I, :J => J, :V => V])
    CSV.write("M.csv", df)
end

julia> M = sprand(10, 10, 0.2)

julia> save_sparse_matrix(M)
I = [1, 7, 5, 9, 10, 1, 3, 4, 6, 3, 4, 9, 3, 7, 10, 5, 6, 7, 9, 10, 7]

julia> I
UniformScaling{Bool}
true*I

Thanks. :smiley: