Save and load data for interactive analysis (why is JLD2 @load "deprecated"?)

New to Julia
1

During interactive data analysis, I sometimes want to save some variables to disk, and then load them again later to continue. For what I want to do, this works perfectly (using JLD2):

# Do first part of analysis
jldsave("part1.jld2"; var1, var2, var3)
# Perhaps in another file do some other analysis
jldsave("part2.jld2"; var4)
# Etc.
# In another file, load all results from part1 and part2
@load "part1.jld2"
@load "part2.jld2"

For my use case, @load is very convenient, more convenient than having to list all variables I want to read as indicated in JLD2 documentation with a combination of jldopen and read.
It is also more convenient than getting all variables in a Dict with load.
The problem is that the JLD2 documentation lists @load in the Legacy section, which I interpret as “deprecated” (so, for me trying to learn, it is interpreted as “don’t bother, this might not even work anymore in the future”).

The doc says @load and @save “unnecessarily introduce new macro-based syntax”, but I cannot find how to get the same results differently, so I struggle to understand why it is “unnecessary”.

Is there another way to get all variables in a JLD2 file (as variables, not in a Dict) without having to list them one-by-one?
This seems to have been discussed before, but I have only found rather old answers so I thought it would be better to ask again.

For completeness, I thought about a workaround by getting a Dict and dumping all keys as variables with their values, which is explained here, but that is also a quite old thread and it certainly doesn’t seem “cleaner” than using @load.

2

Hi @mjmat ,

I understand that @load "file.jld2" can be very useful for you here.
The issue that I have with this is that it is brittle:

  • The file has to be available at “parse time” → JLD2 will try to load the file while trying to expand the macro in order to find all the required variable names
  • The macro will try to create or overwrite variables in your session, depending on what is in the file, which makes it difficult to reason about.
  • It cannot work in package code
  • The provided file name has to be a string literal. (So the file can be opened at parse time)

Short answer: No, there is no other recommended way because I don’t think it is a good idea to do this in the first place.
I moved it to Legacy because I don’t want to promote its usage. It is unlikely that it will be removed entirely any time soon.

Of course, I also have similar requirements at work sometimes. This is how I do it:

# compute code with output var1 var2, var3
computedata = (; var1, var2, var3)
jldsave("part1.jld2"; computedata)

##
data = load("part1.jld2")
# Either use computedata.var1 or use:
(; var1, var2, var3) = data
# Or if analysis is wrapped into a function:
function analysisfun(; var1, var2, var3)
      # like so
end

analysisfun(; data...)
3 Likes
3

Thank you for taking the time to provide a detailed answer.
I understand that @load is not a good idea, but sometimes it is useful so I am glad it’s not going away “too soon”.
And I will try to use the method you suggest, putting everything in a named tuple is a neat trick, I really like your suggestion to use computedata.var1 etc in your solution. Thanks again!

1 Like