# Leveraging the GDAL VSI for “raw” access to NetCDF/HDF5 files inside a Zip archive?

**URL:** <https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129>\
**Category:** General Usage\
**Tags:** question, ccall, packages, hdf5, gdal\
**Created:** [April 16, 2025, 3:38pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129 "2025-04-16T15:38:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Oblomov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oblomov/32/211073_2.png) [@Oblomov](https://discourse.julialang.org/u/Oblomov)\
**Post date:** [April 16, 2025, 3:38pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/1 "2025-04-16T15:38:20Z")

</div>

Hello all, I have an interesting conundrum.

My situation is the following: I have some data which is distributed as a zip archive of .nc files. The files are rather large, so I would prefer to not unzip the archive to read them. Since I need to read these .nc files with GDAL, I can leverage the GDAL VSI to access `"/vsizip/$(zip_file)/$(nc_file)"` and read the raster bands without issues.

_However_ some of the data in these files is not exposed by GDAL, so I’m using the `ZipFile` and `HDF5` packages to load the .nc file into memory and then process them, with something like

```julia
    r = ZipFile.Reader(zip_name)
    for f in r.files
        if f.name == h5_name
            bytes = read(f)
            h5open(bytes, "r"; name="MEM:$(zip_name)/$(h5_name)") do fid
                ret = get_all_hdf5_metadata(fid)
            end
        end
    end
    close(r)

```

This works, but is honestly somewhat consuming (in time and memory) since I have to read the entire .nc file into memory before processing.

GDAL itself provides what HDF5 calls a VFD to make it leverage GDAL’s own VSI to access the file(s). This is “relatively” straightforward to leverage in C(++) as long as one has everything at hand, by calling GDAL’s provided [`HDF5VFLGetFileDriver()`](https://github.com/OSGeo/gdal/blob/master/frmts/hdf5/hdf5vfl.h), which registers the VFD with HDF5, and then pass to `H5Fopen` an `H5P` (property) with the driver set to the mentioned driver, as done in [`GDAL_HDF5Open()`](https://github.com/OSGeo/gdal/blob/master/frmts/hdf5/hdf5dataset.cpp).

So my obvious question would be: would it be possible to achieve this in Julia, assuming I have the respective packages installed? I assume the answer is “yes” provided one calls straight to C/C++?

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [April 16, 2025, 4:14pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/2 "2025-04-16T16:14:29Z")

</div>

> [@Oblomov](#):
>
> _However_ some of the data in these files is not exposed by GDAL

Could you show an example? I mean, some SUBDATASET that `gdainfo` shows but that you can’t access from Julia.

Note, `nc` supports high level of compression and those files gain nothing in being zip’d.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [April 16, 2025, 4:22pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/3 "2025-04-16T16:22:00Z")

</div>

You can use ZipArchives.jl to get a view of an entry in an archive if that is helpful. From what I can tell, HDF5 cannot open views, so you might need to do something hacky with `GC.@preserve` and `unsafe_wrap`.

Here is an example using JSON3. Importantly, this only works because `zip_iscompressed(r, entry)` returns false, meaning the entry is not compressed in the archive. If the entry is compressed, things get a lot more complicated.

```julia-repl
julia> using ZipArchives, Mmap, JSON3

julia> r = ZipReader(mmap(open("example.zip")));

julia> entry = something(zip_findlast_entry(r, ".zattrs"))
2

julia> start = firstindex(parent(r)) + zip_entry_data_offset(r, entry)
132

julia> stop = start + Int(zip_compressed_size(r, entry)) - 1
245

julia> @assert !zip_iscompressed(r, entry)

julia> JSON3.read(view(parent(r), start:stop))
JSON3.Object{SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Vector{UInt64}} with 3 entries:
  Symbol("time (s)") => 150.0
  :version => "0.8.0"
  :uuid => "37eee81f-88ae-4d11-b6b3-d38e1ccf0a08"

```

---

<div class="post-metadata">

**Author:** ![Oblomov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oblomov/32/211073_2.png) [@Oblomov](https://discourse.julialang.org/u/Oblomov)\
**Post date:** [April 16, 2025, 4:59pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/4 "2025-04-16T16:59:06Z")

</div>

> [@joa-quim](#):
>
> Could you show an example? I mean, some SUBDATASET that `gdainfo` shows but that you can’t access from Julia.

The files in question are EUMETSAT’s data files for the new FCI sensor mounted on MTG satellites. The information I’m interested in is stored in 0-dimensional (scalar) datasets that are ignored by GDAL (if you try to open them explicitly with `gdal`, which you can when you know the HDF5 path to the dataset, it will explicitly error out with the information about them not being supported).

> [@](#):
>
> Note, `nc` supports high level of compression and those files gain nothing in being zip’d.

True, but how these files are generated it outside of my control 🕶 Still, in this case there’s about a 10% reduction which is around 100MB per file. And since they are compressed, I’m afraid I can’t even use @nhz2’s trick 8-(

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [April 16, 2025, 5:03pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/5 "2025-04-16T17:03:52Z")

</div>

Do you have a direct link to one of such files?

GMT.jl has a `gdalinfo` function that returns a string with the same info as the CLI `gdalinfo`. Maybe you can parse that string to extract what you need.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [April 16, 2025, 5:19pm UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/6 "2025-04-16T17:19:53Z")

</div>

And maybe [MetopDatasets.jl](https://github.com/eumetsat/MetopDatasets.jl) can help too

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [April 17, 2025, 1:43am UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/7 "2025-04-17T01:43:36Z")

</div>

Not sure if this will help, but it was a lightbulb moment for me when facing the problem of extracting one file from a very large remote Zip file. If the server for the files supports range requests (which any non-toy server does these days), you can download only the particular HDF5 file you want by reading the directory at the end of the Zip file to get the start and end in bytes of the compressed file you want. Then you use another range request to download and then uncompress that file.

I’ve recently coded such a file extractor for my own purposes [here](https://github.com/jamesrhester/ImgCIFHandler.jl/blob/main/src/zipfilemanager.jl). The `ZipArchive` constructor reads the directory (downloading the last 65000 bytes) and the resulting instance is then passed as an argument to download particular images specified by their path in the archive. I’ve tested this on Zenodo, whose server would only admit to doing range requests if you actually tried one.

---

<div class="post-metadata">

**Author:** ![Oblomov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oblomov/32/211073_2.png) [@Oblomov](https://discourse.julialang.org/u/Oblomov)\
**Post date:** [April 17, 2025, 5:46am UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/8 "2025-04-17T05:46:39Z")

</div>

@joa-quim I’m afraid I’m not free to share the data directly, but it can be downloaded from the EUMETSAT Data Portal at [EUMETSAT - Data Store](https://data.eumetsat.int/data/map/EO:EUM:DAT:0662)

I appreciate the recommendations, but as I said GDAL does not present that data in any way (currently; I have opened an issue on GitHub about it). `gdalinfo` in particular doesn’t show it. (METOP is a completely different file format.)

---

<div class="post-metadata">

**Author:** ![Oblomov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oblomov/32/211073_2.png) [@Oblomov](https://discourse.julialang.org/u/Oblomov)\
**Post date:** [April 17, 2025, 5:49am UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/9 "2025-04-17T05:49:45Z")

</div>

> [@frtps](#):
>
> Not sure if this will help, but it was a lightbulb moment for me when facing the problem of extracting one file from a very large remote Zip file.

The trick you suggest is quite interesting, but I’m afraid it doesn’t apply in my case. We want to process the content of the whole archive, so we do need to get the whole .zip anyway. But we’d rather avoid unpacking it if possible (which we can do insofar as we only need the GDAL-accessible information from the .nc files contained within, and also if we want the rest —at the expense of time and memory).

---

<div class="post-metadata">

**Author:** ![Oblomov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oblomov/32/211073_2.png) [@Oblomov](https://discourse.julialang.org/u/Oblomov)\
**Post date:** [April 17, 2025, 6:27am UTC](https://discourse.julialang.org/t/leveraging-the-gdal-vsi-for-raw-access-to-netcdf-hdf5-files-inside-a-zip-archive/128129/10 "2025-04-17T06:27:47Z")

</div>

OK I’ve been looking into `ccall()` and this would do what I want _if the symbols I wanted to access were exported by the gdal library_. But they are not, so it seems I’m stuck in the current situation 8-(