Can't read old JLD2 file

Tooling
10

I disagree with this — I think the best solution is to support datatypes which have a native or near-native correspondence with what is supported by HDF5, and require explicit conversion by the user for the rest.

This seems like an unnecessary overhead, but making the mapping automatic is not a strategy that works well even in the medium run, as such mappings inevitable break down, sometimes silently (with no error, just corrupted data).

IMO this has been the source of a lot of grief for users who thought that they can just use JLD2 or similar as a fast, native, and reasonably robust data storage format. I think that there are two viable approaches:

  1. emphemeral data one can regenerate at little cost: Serialization.serialize and friends or Mmap.mmap, depending on the details,

  2. structured data that is costly to regenerate: map to something that is supported by HDF5, and for long-term storage write a lot of relevant metadata, including strings with explanations, and hope for the best.

2 Likes