JLD2: triggering the actual writes to file

Specific Domains Data
1

Let’s say I saved a dataset named “dataset” to “file.jld2”:

using JLD2, Random
jldopen("file.jld2", "w") do fd
    fd["dataset"] = [randn(100,100) for i=1:100]
end

Now, I want to alter one of the entries in the dataset:

jldopen("file.jld2", "a") do fd
    ds = fd["dataset"]
    ds[1] = randn(100,100)
end
  • When is the actual write to the file triggered in the last case? When I do the assignment ds[1] = randn(100,100)? Or when the file is closed?
  • Does it depend on the memory size of the element I am trying to write? I mean, smaller or larger than buffer.
2

Hi @Gregstrq,

the "a" really means append. Arrays are written immediately after assignment
and it is currently not possible to edit them after that.

This is a limitation of JLD2 (lack of implementation). In principle it might be possible. (Try editing a JLD2 file using the HDF5 libary)

3

Do you know when the writes are triggered in the case of HDF5?

4

This usually also happens immediately.

https://juliaio.github.io/HDF5.jl/stable/#Memory-mapping