Leveraging the GDAL VSI for “raw” access to NetCDF/HDF5 files inside a Zip archive?

Oblomov
Wed, 16 Apr 2025 15:38:20 +0000
General Usage

Hello all, I have an interesting conundrum.

My situation is the following: I have some data which is distributed as a zip archive of .nc files. The files are rather large, so I would prefer to not unzip the archive to read them. Since I need to read these .nc files with GDAL, I can leverage the GDAL VSI to access "/vsizip/$(zip_file)/$(nc_file)" and read the raster bands without issues.

However some of the data in these files is not exposed by GDAL, so I’m using the ZipFile and HDF5 packages to load the .nc file into memory and then process them, with something like

    r = ZipFile.Reader(zip_name)
    for f in r.files
        if f.name == h5_name
            bytes = read(f)
            h5open(bytes, "r"; name="MEM:$(zip_name)/$(h5_name)") do fid
                ret = get_all_hdf5_metadata(fid)
            end
        end
    end
    close(r)

This works, but is honestly somewhat consuming (in time and memory) since I have to read the entire .nc file into memory before processing.

GDAL itself provides what HDF5 calls a VFD to make it leverage GDAL’s own VSI to access the file(s). This is “relatively” straightforward to leverage in C(++) as long as one has everything at hand, by calling GDAL’s provided HDF5VFLGetFileDriver(), which registers the VFD with HDF5, and then pass to H5Fopen an H5P (property) with the driver set to the mentioned driver, as done in GDAL_HDF5Open().

So my obvious question would be: would it be possible to achieve this in Julia, assuming I have the respective packages installed? I assume the answer is “yes” provided one calls straight to C/C++?

10 posts - 4 participants