I am working on a simulation that stores and remove values (some form of float) at various points in a 3-dimensional space, represented by an array with a size of ~(2e4 x 4e4 x 1e2). Specifically, it records the property of the sediments (e.g. the average size of the sand grains) that get deposited or eroded from an ocean basin with uneven sea-bed surface elevation (measured from some uniform “base” level, which forms the bottom layer of the 3-D array, with “depth” being the shortest dimension). Aside: There might be a need to have additional arrays like this to store other relevant values.
Most of these deposition/erosion events happen in the central (i.e., not near the edges) part of this array, at seemingly random points (depending on the overlying water current).
With an array this size, memory will be overwhelmed if I use a simple 3-D array (right?). I would like to hear your suggestions on what best to do. So far, I have considered:
(i) Using a collection of sparse matrices, each one being a “slice” of the 3-D grid along the length or width. (The slicing is because I could not find a sparse-array implementation in Julia that is not limited to 2-D.) The issue with this is how best to automatically allocate the correct number of slices at the beginning of the simulation, when the length and width of the ocean basin gets specified, and how to reference/track them efficiently as the simulation progresses. How to write sparse arrays to file(s) without converting them back into full arrays is also a question that I have.
(ii) Using something like the Zarr implementation (but adapted to Julia, of course). My concern about this is whether dividing the array into compressed “chunks” that are slices of the 3-D array is going to help with the speed of accessing random points in the central part of this 3-D grid, given the additional computational time needed to split and compress these “chunks”.
I would love to hear any critique and comments on what I have considered above, and any suggestions or ideas that you might have.