Hard Disk storage solution for a (large) array of matrices

Mouad
Thu, 01 Jun 2023 15:46:25 +0000
Performance

I just finished an enormous scientific simulation (billions of regressions, trillions of allocations).
The final output are a few arrays of matrices: 100000-element Vector{Matrix{Float64}}. I am looking for a solution to store them locally to recall them later in Julia. (both for future use and because I need to kill julia as it’s boggling up 7gb of RAM and 95% GPU after all the heavy lifting).

Essentially, how do I store structures of the type:

A= [ rand(3,3) for i in 1:10^7 ]

Cursory browsing of binary formats packages (HDF5, JLD2 , etc) yields no results.
I am really married to the Array{Array{Float64, 2}, 1} struct as 1000s of lines of code are written around it. I suppose I’ll have to reshape them into vectors for storage to hcat() them after recall? If so, what’s an efficient algorithm to do so please?
Thanks!

13 posts - 4 participants