# Appending an element to a JLD2 file

**URL:** <https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424>\
**Category:** General Usage\
**Tags:** question, hdf5, jld2\
**Created:** [October 13, 2017, 3:54pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424 "2017-10-13T15:54:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [October 13, 2017, 3:54pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424/1 "2017-10-13T15:54:33Z")

</div>

I’m curious what the right way to append elements to a vector stored in a JLD2 file, assuming that I want to save the data to disk every time I add an element (because each element is expensive to compute).

The first thing I tried was:

```julia
jldopen("test.jld2", "a+") do file
    file["x"] = []
end

for i in 1:10
    jldopen("test.jld2", "a+") do file
        push!(file["x"], 1)
    end
end

```

but that actually results in the file being empty, because doing `file["x"]` gives me a copy of the data, and appending to that copy does nothing to the data on disk.

So, instead, I could do:

```julia
results = []
for i in 1:10 
    push!(results, 1)
    jldopen("test.jld2", "w") do file
        file["x"] = results
    end
end

```

but this will write the entire `results` vector to disk at every iteration, which seems wasteful.

Am I missing something obvious? Is there a better way?

---

<div class="post-metadata">

**Author:** ![ggggggggg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ggggggggg/32/265_2.png) [@ggggggggg](https://discourse.julialang.org/u/ggggggggg)\
**Post date:** [October 13, 2017, 4:31pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424/2 "2017-10-13T16:31:20Z")

</div>

HDF5 (on which JLD is based) is not very efficient with small writes, and by default leaves you open to losing data if your program crashes before flushing. The new single writer multiple reader (SWMR) feature allows you to flush a specific dataset, and I believe it syncs the dataset and metadata to disk. I think if you use SWMR, or if you just flush the file after every write, you will be safe.

Use `HDF5.jl` directly to create a dataset with [extendible dimensions](https://github.com/JuliaIO/HDF5.jl/blob/master/doc/hdf5.md) (search for extendible), and push to that. The SWMR API is not in the `HDF5.jl` docs, but the [`swmr.jl`](https://github.com/JuliaIO/HDF5.jl/blob/master/test/swmr.jl) test file shows the usage.

---

<div class="post-metadata">

**Author:** ![ggggggggg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ggggggggg/32/265_2.png) [@ggggggggg](https://discourse.julialang.org/u/ggggggggg)\
**Post date:** [October 13, 2017, 4:34pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424/3 "2017-10-13T16:34:29Z")

</div>

Also both of you examples show opening and closing the JLD file on each iteration, which is expensive. You might consider writing to a flat binary file on each iteration as a backup, and just writing the complete results in JLD at the end. Or

```julia
jldopen("test.jld2", "w") do file
results = []
    for i in 1:10 
        push!(results, 1)
        file["x$i"] = results[i]
    end
end

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [October 13, 2017, 8:06pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424/4 "2017-10-13T20:06:17Z")

</div>

Ok, makes sense. Thanks!

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [October 20, 2017, 8:06pm UTC](https://discourse.julialang.org/t/appending-an-element-to-a-jld2-file/6424/5 "2017-10-20T20:06:55Z")

</div>

x-ref: [HDF5 speed?](https://discourse.julialang.org/t/hdf5-speed/6584)