How to compress HDF5 data?

General Usage
,
1

How to compress HDF5 data ?

julia> using HDF5

julia> fid=h5open("file.h5","w")
HDF5 data file: file.h5

julia> close(fid)

julia>

julia> fid = h5open("file.h5","r+")
HDF5 data file: file.h5

julia> write(fid,"A",rand(5,5),"compress")
ERROR: MethodError: no method matching d_create(::HDF5.HDF5File, ::String, ::HDF5.HDF5Datatype, ::HDF5.HDF5Dataspace, ::
String)
Closest candidates are:
  d_create(::Union{HDF5.HDF5File,HDF5.HDF5Group}, ::String, ::HDF5.HDF5Datatype, ::HDF5.HDF5Dataspace, ::String, ::Any,
::Any...) at C:\Users\PC\.julia\v0.5\HDF5\src\plain.jl:828
  d_create(::Union{HDF5.HDF5File,HDF5.HDF5Group}, ::String, ::HDF5.HDF5Datatype, ::HDF5.HDF5Dataspace) at C:\Users\PC\.j
ulia\v0.5\HDF5\src\plain.jl:822
  d_create(::Union{HDF5.HDF5File,HDF5.HDF5Group}, ::String, ::HDF5.HDF5Datatype, ::HDF5.HDF5Dataspace, ::HDF5.HDF5Proper
ties) at C:\Users\PC\.julia\v0.5\HDF5\src\plain.jl:822
  ...
 in _d_create(::HDF5.HDF5File, ::String, ::Array{Float64,2}, ::String, ::Vararg{String,N}) at C:\Users\PC\.julia\v0.5\HD
F5\src\plain.jl:1545
 in d_create(::HDF5.HDF5File, ::String, ::Array{Float64,2}, ::String) at C:\Users\PC\.julia\v0.5\HDF5\src\plain.jl:1552
 in _d_write(::HDF5.HDF5File, ::String, ::Array{Float64,2}, ::String, ::Vararg{String,N}) at C:\Users\PC\.julia\v0.5\HDF
5\src\plain.jl:1578
 in write(::HDF5.HDF5File, ::String, ::Array{Float64,2}, ::String) at C:\Users\PC\.julia\v0.5\HDF5\src\plain.jl:1614

julia> write(fid,"A",rand(5,5),'compress')
ERROR: syntax: invalid character literal

Paul

2

You can’t use the write function to define the compression, you need to create chunks.

Have a look at the docs of HDF5.jl here: https://github.com/JuliaIO/HDF5.jl/blob/master/doc/hdf5.md

3

HDF5 works pretty good with Blosc.jl
You can use multithreaded compression using Blosc.jl

1 Like