Obviously a corporation like Mathworks (for all its limitations) shines in its ability to define an official serialization interface: .mat (complexity of .mat versions aside)
Is there any plan to do the same for JLDx? eg “everybody just use JLD2 from now on”.
Just read this, now I’m even more confused, lol
Sometimes its good to have an open community building a ton of alternatives, sometimes it’s not as good. I think this is a place where it hasn’t turned out so good and someone needs to sit down and consolidate it. The summary right now is:
- Julia’s serialization is the best, but cannot necessarily cross versions.
- HDF5 is stable and won’t change, but that means it supports the basics (arrays and such), and that’s pretty much what it will do.
- JLD2 is great and got new maintainers. That might mean it’s the thing to use from now on, but I haven’t tried it since it got the new crew.
- BSON has some architectural issues and is in a maintenance state where it won’t change much.
- JSON is pretty nice if it works for the types you’re using.
- Anything else I wouldn’t consider, or at least, it’s “use at your own risk” levels of maintenance and use by the community.
I’d probably just stick to JLD2 from now on.
Would it be possible to write a cross-version deserialization package using the new libjulia_jll’s?
That is a fantastic question…
JLD2 is great, especially with new maintainers.
With one caveat: unless you can lock all package versions in a Manifest.toml file, avoid serializing types that you don’t own or aren’t in base. Package updates can change things that will break your .jld2 file.
If you want to store in binary and reliability, long-term restorability, and, perhaps, cross language readability are important in your context I would go with HDF5. Basically every programming language supports it and although it only supports “basic objects” I never found this to be a big issue.
Actually, I think of the need to convert to “basic” types (strings, numbers, arrays of them) as a feature for medium- to long-term data storage.
Anything more complex is coupled too closely to the current state of packages (and in the medium run, Julia) for comfort.