Recommended serialization interface in Oct 2020: JLD, JLD2,

New to Julia
, ,
1

Obviously a corporation like Mathworks (for all its limitations) shines in its ability to define an official serialization interface: .mat (complexity of .mat versions aside)

Is there any plan to do the same for JLDx? eg “everybody just use JLD2 from now on”.

-edit-
Just read this, now I’m even more confused, lol

2

Sometimes its good to have an open community building a ton of alternatives, sometimes it’s not as good. I think this is a place where it hasn’t turned out so good and someone needs to sit down and consolidate it. The summary right now is:

  • Julia’s serialization is the best, but cannot necessarily cross versions.
  • HDF5 is stable and won’t change, but that means it supports the basics (arrays and such), and that’s pretty much what it will do.
  • JLD2 is great and got new maintainers. That might mean it’s the thing to use from now on, but I haven’t tried it since it got the new crew.
  • BSON has some architectural issues and is in a maintenance state where it won’t change much.
  • JSON is pretty nice if it works for the types you’re using.
  • Anything else I wouldn’t consider, or at least, it’s “use at your own risk” levels of maintenance and use by the community.

I’d probably just stick to JLD2 from now on.

3

Would it be possible to write a cross-version deserialization package using the new libjulia_jll’s?

4

That is a fantastic question…

5

JLD2 is great, especially with new maintainers.

With one caveat: unless you can lock all package versions in a Manifest.toml file, avoid serializing types that you don’t own or aren’t in base. Package updates can change things that will break your .jld2 file.

6

If you want to store in binary and reliability, long-term restorability, and, perhaps, cross language readability are important in your context I would go with HDF5. Basically every programming language supports it and although it only supports “basic objects” I never found this to be a big issue.

7

Actually, I think of the need to convert to “basic” types (strings, numbers, arrays of them) as a feature for medium- to long-term data storage.

Anything more complex is coupled too closely to the current state of packages (and in the medium run, Julia) for comfort.