[ANN] SmallZarrGroups.jl - Save and load hierarchy of arrays and metadata

nhz2 · March 21, 2023, 2:36pm

Hello all,

I was looking for a way to save snapshots of my simulation state to disk for restart, archival, and analysis purposes. I wanted these snapshots to be easily readable from python. Lastly, I didn’t care about write performance, I was only saving about 1 MB per snapshot, but I wanted a format that was efficient with disk space and could be quickly read in multi-threaded analysis scripts.

This lead me to the zarr format and Zarr.jl and
zarr-python.

Zarr.jl is a great package, however, the main issue I have with it is that it doesn’t have a 1.0 release and is missing zip file support.

To solve my issues, I created a zarr reading and writing package with a much narrower scope than Zarr.jl:

I would greatly appreciate feedback on the current API before it becomes fixed when version 1.0 is released.

The following tests show examples of the current API:

github.com

medyan-dev/SmallZarrGroups.jl/blob/main/test/test_simple-usage.jl

using SmallZarrGroups
using DataStructures: SortedDict, OrderedDict
using Test

@testset "create empty ZGroup" begin
    zg = ZGroup()
    @test repr("text/plain",zg) == """
    📂\
    """
end

@testset "copy data to a group" begin
    zg = ZGroup()

    # Save and copy an abstract array with setindex!
    zg["random_data"] = rand(30,10)
    @test repr("text/plain",zg) == """
        📂
        └─ 🔢 random_data: 30×10 Float64 \
        """

This file has been truncated. show original

Nathan