Saving sparse matrix

How to save a sparse matrix in Julia? I want the saved file to be small since my Matrix size is about (2^26,2^26).

I would just save nonzero indexes and values as CSV:

using SparseArrays, DataFrames, CSV
M = sprand(10, 10, 0.2)
I, J, V = findnz(M)
df = DataFrame([:I => I, :J => J, :V => V])
CSV.write("/tmp/spmatrix.csv", df)

Thank you, the problem is that CSV file is very large, for example in python I use *.npz which is 2-5 MB but here it can go up to 500 MB.

I don’t know this format, but you can also compress the CSV, eg with gzip.

Alternatively, you can save a bit of space by using the CSC representation (ie export directly from the fields), but I don’t think it is worth it.

1 Like

Great answer. Minor remark in passing: with LinearAlgebra.I, I is no longer a great choice in I, J, V. :disappointed_relieved: R, C, V or M, N, V are viable alternatives I’ve seen used.

For production code I would wrap it in a function so this should not matter.

1 Like

Very true!

using SparseArrays, DataFrames, CSV, LinearAlgebra
function save_sparse_matrix(M)
    I, J, V = findnz(M)
    println("I = ", I)
    df = DataFrame([:I => I, :J => J, :V => V])
    CSV.write("M.csv", df)

julia> M = sprand(10, 10, 0.2)

julia> save_sparse_matrix(M)
I = [1, 7, 5, 9, 10, 1, 3, 4, 6, 3, 4, 9, 3, 7, 10, 5, 6, 7, 9, 10, 7]

julia> I

Thanks. :smiley: