Loading entire jld2 file

If I’ve saved my data to a jld2 format file using FileIO, is there a quick way to just load in the entire file at once? I seem to have to name all the entries in the file one by one.

I think it is just @load file.

You may have seen this, but the package page now says:

WARNING : This package is not actively maintained and has several known bugs that are unlikely to be resolved in the near future. Please consider using Julia’s built-in serializer if you need performance, or the previous JLD package if you need HDF5 compatibility.

I hadn’t noticed that. Is there a tutorial anywhere on using the serializer as a replacement for JLD2? Alternatively, is there a way to just use FileIO to handle this?

Alternatively, what’s the recommended way to save a bunch of variables to disk now? I am not specifically wedded to HDF5 based tools, though they certainly have advantages.

Maybe you can define your own struct that contains all of your variables, and then just save/load that 1 variable. Not sure if this is sensible…

There isn’t one recommended way. Lots of people ask that question, and you can find lots of answers if you search discourse for threads about saving / loading user defined types. The package to use depends on the type of data that you plan to store.

One recommendation that I have seen a few times and that I am beginning to come around to: write your own converter from your user defined types to something easy to serialize (e.g. a Dict) and then store that in a format that is robust (e.g. JSON).

I have not used the serialize / deserialize approach because those files can become unreadable at any time. They are meant for short term storage only.

3 Likes

Does it appeare the JLD module have continued support? I can’t remember what the reason was, but for some reason they came out with JLD2. Was it because JLD depended on HDF5 library or something?

Is ur data tabular? If it is these r your options Unable to save Unitful unit into a JLD file

I have been thinking: perhaps the problem is not nearly as hard as it looks.
The hard part is constructors. We are asking our code to load a Foo{T} struct and then figure out how to construct that object.
At least in my applications, I know how to construct the object that I’m trying to load. Moreover, pretty much everything that I am trying to save and load is a DataTypes object (in the sense of StructTypes.jl).
I think I could cover 95% of my use cases (especially the most tricky ones where I try to save nested user defined types) with the simple approach of:

  1. Recursively convert the object to a Dict{:Symbol, simple known types}
  2. Save that to a robust format, say JSON.
  3. When loading, provide a randomly initialized, fully constructed Foo{T} object. Recursively walk the loaded Dict and copy values into Foo.

This only works when everything is mutable, but that would be a price I would be willing to pay. Am I missing a reason why this would be a bad idea?

What makes you say that? When you deserialize something you don’t run the constructor, you just directly create the type from the data.

My data is a couple of arrays (which could be large) along with some scalar parameters. The JLD/JLD2 file formats worked pretty well for this.

In part, I am basing this on my experience with JLD2. If I define a Foo object in some module A and later load it, I need to issue using A in Main. Otherwise, my understanding is that JLD2 returns its own version of a Foo object (not an A.Foo).

More generally, when I thought about how to actually write code that saves / loads the structs that I define, I could not figure out how to call the constructor correctly. As an example, if I try to load an A.Foo that contains a B.Bar, I would have to bring A and B into the namespace and then determine (from propertynames?) how to call the constructors (recursively).

Things get more complicated when the types are parameterized.