[JLD2] How to save an obj on file, replacing that object but not the whole file

General Usage
1

In JLD2 I would like to save an object to a sort of “library file”, replacing the same object name if it exists, but keeping the other object if they exists.
The problem is that even with append JLD2 returns an error if the object already exists in the file instead of replacing it:

using JLD2

model1 = 1
model2 = 2
jldopen("modelslib.jld2", "a+") do file
    write(file, "model1", model1)
end
jldopen("modelslib.jld2", "a+") do file
    write(file, "model2", model2)
end
model1j = load("modelslib.jld2", "model1")
model2j = load("modelslib.jld2", "model2")
model1 = 10
jldopen("modelslib.jld2", "a+") do file
    write(file, "model1", model1)
end # error, foo1 exists (I want instead it replaced)
2

Have you tried jldsave or JLD2.save_object?

https://juliaio.github.io/JLD2.jl/dev/#JLD2.save_object

Is the data that you are trying to write exactly the same size? I’m wondering if we can just write new data into an existing dataset or if we need to delete the dataset and replace it.

A JLD2 file just a HDF5 file, so you should be able to use HDF5.jl to modify it if necessary. Depending on what you do JLD2.jl may not be able to read it though.

3

Yes, I have tried them but they overwrite the file.
It seems there is no way to “update” only one object (aside of loading all the objects and resaving the desired ones).

4

Ah, perhaps this will help:

julia> using JLD2

julia> A = rand(16); B = rand(16);

julia> jldsave("example.jld2"; A)

julia> jldopen("example.jld2", "r") do f
           println(keys(f))
       end
["A"]

julia> jldopen("example.jld2", "r+") do f
           f["B"] = B
       end
16-element Vector{Float64}:
 0.5608959073449324
 0.3638963529780811
 0.07184337169387223
 0.18897128618321324
 0.66329261122493
 0.5220036617458721
 0.4285742412772173
 0.6509469598307095
 0.7187628817194093
 0.5203396168881611
 0.4418571079136371
 0.9003617232080112
 0.2346391079104828
 0.9180306350902839
 0.5155522362415368
 0.6388304659727359

julia> jldopen("example.jld2", "r") do f
           println(keys(f))
       end
["A", "B"]

julia> fill!(A, 0)
16-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

julia> C = zeros(256, 256);

julia> jldopen("example.jld2", "a+") do f
           f["C"] = C
       end;

julia> jldopen("example.jld2", "a+") do f
           f["A"] .= A
       end;

julia> jldopen("example.jld2", "a+") do f
           println(keys(f))
       end
["A", "B", "C"]
5

You haven’t “updated” the object on the file:

julia> Aj = load("example.jld2", "A")
16-element Vector{Float64}:
 0.5740161069259119
 0.23376757546050675
 0.13016224315460012
 0.5986203452744622
 0.0014843901260015446
 0.08085386257112082
 ⋮
 0.6432398769774624
 0.3902720389879373
 0.2968135277921722
 0.199786865259862
 0.6854068247070402
6

You’re right. I guess we’ll have to go with my first idea for now.

using HDF5

julia> h5open("example.jld2", "r+") do f
           A = f["A"][] .= 0
           f["A"][:] = A
       end
16-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

julia> jldopen("example.jld2", "r+") do f
           f["A"]
       end
16-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
2 Likes
7

The other idea is to delete! the prior “variable”.

julia> using JLD2

julia> A = rand(16); B = rand(16);

julia> jldsave("example.jld2"; A, B)

julia> jldopen("example.jld2", "a+") do f
           delete!(f, "A")
           f["A"] = zeros(16)
       end
16-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

julia> jldopen("example.jld2", "a+") do f
           f["A"]
       end
16-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
2 Likes
8

thanks. Indeed it works even for complex different objects:

julia> using JLD2

julia> struct Foo
           x::Int64
       end

julia> struct Goo
           y::String
           z::Function
       end

julia> function model_save(filename;kargs...)
    jldopen(filename, "a+") do f
        for (k,v) in kargs
            ks = string(k)
            if ks in keys(f)
                delete!(f, ks)
            end
            f[ks] = v
        end
    end
end
model_save (generic function with 1 method)

julia> a = Foo(1)
Foo(1)

julia> b = Foo(2)
Foo(2)

julia> model_save("example.jld2";a)
Foo(1)

julia> model_save("example.jld2";b)
Foo(2)

julia> a = Goo("foo",maximum)
Goo("foo", maximum)

julia> model_save("example.jld2";a)
Goo("foo", maximum)

julia> aj = load("example.jld2", "a")
Goo("foo", maximum)