Interpolate over large HDF5 arrays

henry2004y
Tue, 18 Mar 2025 04:17:06 +0000
General Usage

Hi,

I have a extremely large standard HDF5 file (2 TB) which saves a 3D array, let’s say B. With HDF5.jl, I can access individual elements of the array in the same way as this thread shows: load-hdf5-file-larger-than-memory.

using HDF5

fid = h5open(fname, "r")
B = fid["B"] # size(B) = (10000, 10000, 4992), eltype(B) = Float32
B[10,10, 1] # this works

However, when I tried to perform interpolations over B using Interpolations.jl, I got

using Interpolations

itp = extrapolate(interpolate(B, BSpline(Linear())), NaN)

ERROR: MethodError: no method matching interpolate(::HDF5.Dataset, ::BSpline{Linear{Throw{OnGrid}}})
The function `interpolate` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  interpolate(::AbstractArray, ::IT, ::GT) where {IT<:Union{NoInterp, Tuple{Vararg{Union{NoInterp, BSpline}}}, BSpline}, GT<:Union{NoInterp, Tuple{Vararg{Union{NoInterp, Interpolations.GridType}}}, Interpolations.GridType}}
   @ Interpolations ~/.julia/packages/Interpolations/91PhN/src/deprecations.jl:19
  interpolate(::AbstractArray, ::IT) where IT<:Union{NoInterp, Tuple{Vararg{Union{NoInterp, BSpline}}}, BSpline}
   @ Interpolations ~/.julia/packages/Interpolations/91PhN/src/b-splines/b-splines.jl:190
  interpolate(::AbstractArray, ::IT, ::Real, ::Int64) where IT<:Union{NoInterp, Tuple{Vararg{Union{NoInterp, BSpline}}}, BSpline}
   @ Interpolations ~/.julia/packages/Interpolations/91PhN/src/b-splines/b-splines.jl:217

This means that Interpolations.jl expects an AbstractArray as input, but HDF5.Dataset is not an AbstractArray. This array is so large that I cannot possibly read it into memory as a whole. Is it possible to interpolate over such large array?

19 posts - 5 participants