I’m curious what the right way to append elements to a vector stored in a JLD2 file, assuming that I want to save the data to disk every time I add an element (because each element is expensive to compute). The first thing I tried was: jldopen("test.jld2", "a+") do file file["x"] = [] end for…

HDF5 (on which JLD is based) is not very efficient with small writes, and by default leaves you open to losing data if your program crashes before flushing. The new single writer multiple reader (SWMR) feature allows you to flush a specific dataset, and I believe it syncs the dataset and metadata to…

Also both of you examples show opening and closing the JLD file on each iteration, which is expensive. You might consider writing to a flat binary file on each iteration as a backup, and just writing the complete results in JLD at the end. Or jldopen("test.jld2", "w") do file results = [] for i…

Appending an element to a JLD2 file

General Usage

rdeits October 20, 2017, 8:06pm 5

x-ref: HDF5 speed?

Topic		Replies	Views
Adding more data to a JLD2 file General Usage jld2	5	1504	August 5, 2020
Stream into JLD2 file General Usage jld2	8	210	November 22, 2025
H5write analog for JLD General Usage jld	0	629	December 22, 2017
JLD2: triggering the actual writes to file Data jld2	3	484	January 21, 2022
JLD file , how to add new line to existing array in jld file? General Usage	3	562	November 24, 2019

Appending an element to a JLD2 file

Related topics