Loading entire jld2 file

General Usage
1

If I’ve saved my data to a jld2 format file using FileIO, is there a quick way to just load in the entire file at once? I seem to have to name all the entries in the file one by one.

2

I think it is just @load file.

3

You may have seen this, but the package page now says:

WARNING : This package is not actively maintained and has several known bugs that are unlikely to be resolved in the near future. Please consider using Julia’s built-in serializer if you need performance, or the previous JLD package if you need HDF5 compatibility.

4

I hadn’t noticed that. Is there a tutorial anywhere on using the serializer as a replacement for JLD2? Alternatively, is there a way to just use FileIO to handle this?

5

Alternatively, what’s the recommended way to save a bunch of variables to disk now? I am not specifically wedded to HDF5 based tools, though they certainly have advantages.

6

Maybe you can define your own struct that contains all of your variables, and then just save/load that 1 variable. Not sure if this is sensible…

7

There isn’t one recommended way. Lots of people ask that question, and you can find lots of answers if you search discourse for threads about saving / loading user defined types. The package to use depends on the type of data that you plan to store.

One recommendation that I have seen a few times and that I am beginning to come around to: write your own converter from your user defined types to something easy to serialize (e.g. a Dict) and then store that in a format that is robust (e.g. JSON).

I have not used the serialize / deserialize approach because those files can become unreadable at any time. They are meant for short term storage only.

3 Likes
8

Does it appeare the JLD module have continued support? I can’t remember what the reason was, but for some reason they came out with JLD2. Was it because JLD depended on HDF5 library or something?

9

Is ur data tabular? If it is these r your options Unable to save Unitful unit into a JLD file - #10 by xiaodai

10

I have been thinking: perhaps the problem is not nearly as hard as it looks.
The hard part is constructors. We are asking our code to load a Foo{T} struct and then figure out how to construct that object.
At least in my applications, I know how to construct the object that I’m trying to load. Moreover, pretty much everything that I am trying to save and load is a DataTypes object (in the sense of StructTypes.jl).
I think I could cover 95% of my use cases (especially the most tricky ones where I try to save nested user defined types) with the simple approach of:

  1. Recursively convert the object to a Dict{:Symbol, simple known types}
  2. Save that to a robust format, say JSON.
  3. When loading, provide a randomly initialized, fully constructed Foo{T} object. Recursively walk the loaded Dict and copy values into Foo.

This only works when everything is mutable, but that would be a price I would be willing to pay. Am I missing a reason why this would be a bad idea?

11

What makes you say that? When you deserialize something you don’t run the constructor, you just directly create the type from the data.

12

My data is a couple of arrays (which could be large) along with some scalar parameters. The JLD/JLD2 file formats worked pretty well for this.

13

In part, I am basing this on my experience with JLD2. If I define a Foo object in some module A and later load it, I need to issue using A in Main. Otherwise, my understanding is that JLD2 returns its own version of a Foo object (not an A.Foo).

More generally, when I thought about how to actually write code that saves / loads the structs that I define, I could not figure out how to call the constructor correctly. As an example, if I try to load an A.Foo that contains a B.Bar, I would have to bring A and B into the namespace and then determine (from propertynames?) how to call the constructors (recursively).

Things get more complicated when the types are parameterized.