Portability of serialized files

I’m trying to understand what the following actually means from the documentation of julia V1.1.0 about serialization https://pkg.julialang.org/docs/julia/THl1k/1.1.0/stdlib/Serialization.html:

The read-back value will be as identical as possible to the original. In general, this process will not work if the reading and writing are done by different versions of Julia, or an instance of Julia with a different system image.

We have some R applications that are intensively using ‘saveRDS()’ and ‘readRDS()’ for serializing predictive models (=> can’t use HDF5).
I wonder if we could rely on Julia’s ‘serialize()’ and ‘deserialize()’ in the same way.

  • Is there a plan to have some sort of utility tool to upgrade serialized files to make them compatible with a newer version of Julia? with a different system image?
  • What does ‘different system image’ means? that the serialized file can only be read on the same exact machine?
  • Side question: is there a convention for the file extension? .jldb? .jdb?

I’d suggest you avoid serialize() for this. As far as I know

  • There is no near term plan to offer compatibility across versions.
  • A different system image means using a different --sysimage command line option when starting julia. For example, if you wanted to compile frequently used (non-stdlib) packages for faster load time you could create a custom sysimage containing them.
  • Currently reasonable uses for serialize:
    • Caching information which could be recomputed easily when changing julia configuration (version or sysimg).
    • As a wire protocol between cooperating julia processes when using multiprocessing.

As a possible exception, if you have the infrastructure to reliably recompute your model from the data when changing julia versions and you’re willing to package julia with the model itself (or otherwise version the dependency) you could consider using serialize(). Both these things are desirable for robustly developing and deploying models, but I don’t pretend they’re easy :slight_smile:

3 Likes

IIUC this seems like a good use case for BSON.jl or JLD2.jl