How to push! new row to array in hdf5 file?

General Usage
1

How to push! new row to array in hdf5 file without read and write the array?

  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> using HDF5

julia> hfi=h5open("my_file.h5","w")
HDF5 data file: my_file.h5

julia> close(hfi)

julia> fid = h5open("my_file.h5","r+")
HDF5 data file: my_file.h5

julia> write(fid,"A",rand(5,5))

Is somthing l;ike push! ? :

julia> push!(fid,"A",rand(5)) 
ERROR: MethodError: no method matching push!(::HDF5.HDF5File, ::String)
Closest candidates are:
  push!(::Any, ::Any, ::Any) at abstractarray.jl:1715
  push!(::Any, ::Any, ::Any, ::Any...) at abstractarray.jl:1716
  push!(::Array{Any,1}, ::ANY) at array.jl:486
  ...
 in push!(::HDF5.HDF5File, ::String, ::Array{Float64,1}) at .\abstractarray.jl:1715

Paul

1 Like
2

There is set_dims! function to achieve what you need.
However you need to “preallocate” dataset’s dimensions to be able to change them later:

fid = h5open("my_file.h5","w")
A = d_create(fid, "A", Float64, ((5,5),(-1,-1)), "chunk", (5,5))
A[:,:] = randn(5,5)
close(fid)

fid = h5open("my_file.h5","r+")
A = d_open(fid, "A")
set_dims!(A, (6,5)) # add new row
A[6,:] = -1.
close(fid)
2 Likes
3

Thanks, is nice working.
What do this line : ?

|A = d_create(fid, “A”, Float64, ((5,5),(-1,-1)), “chunk”, (5,5))|

Why my code dosent works ?
My code
julia> using HDF5

julia> fid=h5open(“file.h5”,“w”)
HDF5 data file: file.h5

julia> close(fid)

julia> fid = h5open(“file.h5”,“r+”)
HDF5 data file: file.h5

julia> write(fid,“A”,rand(5,5))

julia> close(fid)

julia> fid = h5open(“file.h5”,“r+”)
HDF5 data file: file.h5

julia> A = d_open(fid, “A”)
HDF5 dataset: /A (file: file.h5)

julia> size(A)
(5,5)

julia> set_dims!(A, (6,5)) # add new row
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5D.c line 1167 in
H5Dset_extent(): unable to set extend dataset
major: Dataset
minor: Unable to initialize object
#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dint.c line 2169
in H5D__set_extent(): dataset has contiguous stor
age
major: Invalid arguments to routine
minor: Out of range
ERROR: Error extending dataset dimensions
in h5d_set_extent(::HDF5.HDF5Dataset, ::Array{UInt64,1}) at
C:\Users\PC.julia\v0.5\HDF5\src\plain.jl:2007
in set_dims!(::HDF5.HDF5Dataset, ::Tuple{Int64,Int64}) at
C:\Users\PC.julia\v0.5\HDF5\src\plain.jl:1178

Paul

W dniu 2017-03-12 o 03:05, Tomas Mikoviny pisze:

4

Everything should make sense once you look at HDF5 User’s Guide (5.4.5. Storage Strategies).
Contiguous datasets created with write or d_create without specifying chunked storage are of fixed size => one cannot resize them.
To define chunks and therefore resizable datasets one needs to follow this scheme:
(as in HDF5.jl Mid-level routines)

d = d_create(parent, name, dtype, (dims, max_dims), "chunk", (chunk_dims))
set_dims!(d, new_dims)

Please note max_dims parameter that defines the limits of dimensions of particular dataset. I had called it “preallocation” in earlier post. If one uses max_dims to be (-1,-1), as in example above, you can change any dimension of the dataset freely, as long as it is less then typemax(Uint).

2 Likes