# A future for JLD2?

**URL:** <https://discourse.julialang.org/t/a-future-for-jld2/42958>\
**Category:** Community\
**Tags:** jld2\
**Created:** [July 12, 2020, 7:05pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958 "2020-07-12T19:05:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 12, 2020, 7:05pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/1 "2020-07-12T19:05:00Z")

</div>

Hi everyone,

Julia shines at numerical computing and inevitably we need to write data to disk.  
There are a few binary formats for julia out there such as Serialization, HDF5.jl, MAT.jl, BSON.jl, JLD.jl  
and JLD2.jl that all work but also all have their limitations.  
(speed, file-size, bugs, binary dependencies, stability…)

My personal favourite has always been JLD2 because it is fast, supports compression,  
can store custom structs, and because it is implemented in pure julia.  
Sadly it has not seen any development in recent years and a few bugs have accumulated.  
As of May 29 it has officially been marked as unmaintained.

In my opinion it would be a shame to let JLD2 go which is why I am writing this call for help.

I have already invested some time in understanding the code base and trying to fix some problems.  
See for example:

[https://github.com/JuliaIO/JLD2.jl/pull/198](https://github.com/JuliaIO/JLD2.jl/pull/198)

[https://github.com/JuliaIO/JLD2.jl/pull/197](https://github.com/JuliaIO/JLD2.jl/pull/197)

[https://github.com/JuliaIO/JLD2.jl/pull/196](https://github.com/JuliaIO/JLD2.jl/pull/196)

[https://github.com/JuliaIO/JLD2.jl/issues/195#issuecomment-657185277](https://github.com/JuliaIO/JLD2.jl/issues/195#issuecomment-657185277)

[https://github.com/JuliaIO/JLD2.jl/issues/55#issuecomment-654091106](https://github.com/JuliaIO/JLD2.jl/issues/55#issuecomment-654091106)

One of the biggest road blocks is issue #55. I think that I have made some progress in uncovering the root of the problem but I need help from someone with more knowledge on memory mapping and network file systems (NFS).

Would you be interested in using JLD2 if it gets brought up to speed again?

I would, of course, appreciate any help in terms of coding but also review  
and in particular discussion!

What you think?

---

<div class="post-metadata">

**Author:** ![Paul\_Soderlind](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paul_soderlind/32/1753_2.png) [@Paul\_Soderlind](https://discourse.julialang.org/u/Paul_Soderlind)\
**Post date:** [July 12, 2020, 8:12pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/2 "2020-07-12T20:12:18Z")

</div>

> Would you be interested in using JLD2 if it gets brought up to speed again?

Yes, it would be great if we could rely on JLD2 again.

In case you make any progress, I would be happy to act as a beta tester.

/Paul S

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [July 12, 2020, 11:08pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/3 "2020-07-12T23:08:07Z")

</div>

Thanks for posting this @JonasIsensee. I am also a fan of JLD2, for the same reasons you stated. I would be sad to see it abandoned. Having said that, it would be good to know what some of the folks who have put a lot of time into the juliaIO org think the future holds for saving custom julia structs to disk before really jumping into JLD2. For example, is there a plan for JLD to be maintained for the foreseeable future? If so maybe we should just live with the binary dependency of HDF5 and let development be focused there. Or is it recommended that people use the Serialization std lib for short term storage of custom struct data but use a format that isn’t julia specific for long term storage?

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [July 12, 2020, 11:40pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/4 "2020-07-12T23:40:34Z")

</div>

Personally I’m also still a big fan of JLD2 despite that its maintenance-only at this point, and would love to see it revived. IMO in the top 5 advantages of Julia vs. Python is the ease of serialization, because while Python pickle is super brittle by comparison, essentially everything including arbitrary used-defined types, closures, etc… in Julia are robustly serializable out-of-the-box. Its a bit of a shame to not just absolutely hammer this point by having a great medium-term to-disk serialization option, which, importantly, still at least loads your data even if type definitions change, which can be expected in the medium-term (this is what JLD2 does, modulo bit-rot slowly introducing issues here and there). I don’t know enough to say which librar(ies) it makes most sense to converge on, but it would just be great to have something dependable moving forward.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 13, 2020, 8:29am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/5 "2020-07-13T08:29:48Z")

</div>

I think that _automatic_ externalization for _all_ Julia types in a format that is _stable_ in the long run is a very difficult task because of the complexity of Julia’s type system. It is not necessarily impossible, but takes a lot of work.

You can relax one of the requirements though to get a solution:

1. if stability is not required (ephemeral storage), just serialize with `Serialization`.
2. if simple types are sufficient (arrays, tables, or dictionaries), use one of the well-known formats (CSV, JSON).
3. for more complex types, write a pair of conversion routines to convert to simple types.

It is worth devoting some thought to the actual specs of the problem, and then picking from one of the nice existing libraries. I personally like

> **[GitHub - JuliaData/JSONTables.jl: JSON3.jl + Tables.jl](https://github.com/JuliaData/JSONTables.jl)**
>
> JSON3.jl + Tables.jl. Contribute to JuliaData/JSONTables.jl development by creating an account on GitHub.

> **[GitHub - JuliaIO/HDF5.jl: Save and load data in the HDF5 file format from Julia](https://github.com/JuliaIO/HDF5.jl)**
>
> Save and load data in the HDF5 file format from Julia - GitHub - JuliaIO/HDF5.jl: Save and load data in the HDF5 file format from Julia

> **[GitHub - JuliaData/CSV.jl: Utility library for working with CSV and other...](https://github.com/JuliaData/CSV.jl)**
>
> Utility library for working with CSV and other delimited files in the Julia programming language - GitHub - JuliaData/CSV.jl: Utility library for working with CSV and other delimited files in the J...

> **[GitHub - JuliaData/StructTypes.jl: Abstract definitions and convenience...](https://github.com/JuliaData/StructTypes.jl)**
>
> Abstract definitions and convenience methods for describing, processing, and constructing Julia objects - GitHub - JuliaData/StructTypes.jl: Abstract definitions and convenience methods for describ...

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [July 13, 2020, 8:52am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/6 "2020-07-13T08:52:08Z")

</div>

> [@Tamas\_Papp](#):
>
> I think that _automatic_ externalization for _all_ Julia types in a format that is _stable_ in the long run is a very difficult task because of the complexity of Julia’s type system.

I don’t disagree thats its difficult and certainly can’t really speak to it since I’m not working on it, but JLD2 (and BSON too fwiw) was/is basically already really close to this. In my experience with JLD2, crazy complex things serialize just fine, and if a type changes by the time of deserialization, you get back a ReconstructedType so you can at least still easily get your data out. The fact that it was sooo close to perfect always surprised me when it stopped being actively maintained.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 13, 2020, 8:53am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/7 "2020-07-13T08:53:38Z")

</div>

You are right, some requirements can sometimes be relaxed.  
However not all problems are the same.

In my applications text-based formats such as `CSV` or `JSON` are not an option due to file-size  
and because I don’t want rounding errors just from saving and loading my data.  
An additional difficulty is that makes it hard to work with C libraries is support for `missing`.  
I tend to have large arrays with eltype `Union{Missing, Float64}`.

Another point, of course, is speed.  
And with all those things, as @marius311 already said, JLD2 is/was already really close.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 13, 2020, 8:58am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/8 "2020-07-13T08:58:02Z")

</div>

> [@JonasIsensee](#):
>
> file-size  
> and because I don’t want rounding errors just from saving and loading my data.

> [@JonasIsensee](#):
>
> Another point, of course, is speed.

> **[GitHub - JuliaIO/BSON.jl](https://github.com/JuliaIO/BSON.jl)**
>
> Contribute to JuliaIO/BSON.jl development by creating an account on GitHub.

> [@JonasIsensee](#):
>
> hard to work with C libraries is support for `missing`

Maybe encode it as `NaN`.

---

<div class="post-metadata">

**Author:** ![lhupe](https://avatars.discourse-cdn.com/v4/letter/l/9fc348/32.png) [@lhupe](https://discourse.julialang.org/u/lhupe)\
**Post date:** [July 13, 2020, 9:04am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/9 "2020-07-13T09:04:48Z")

</div>

> [@Tamas\_Papp](#):
>
> Maybe encode it as `NaN` .

this is not optimal. There are applications where `NaN` (i.e. the result is known to not exist) and `missing` (i.e. we don’t know the result) have to be distinguished.

Also, in my experience, switching from JLD2 to BSON slows things down significantly for large files. Moreover, some applications might require saving more than 2 GB of data at once, which also rules out BSON.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [July 13, 2020, 12:10pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/10 "2020-07-13T12:10:35Z")

</div>

Related to the suggestion of writing conversion routines for complex types:

What I find somewhat daunting about that prospect is having to write those converters for deeply nested structs.

My question is: in those cases, a lot of the conversion would just be housekeeping (traversing the nested struct). In my use cases, the actual conversion would usually be trivial (most of the “leaf types” are easily handled by any of the packages used for loading and saving).

Is there a package that handles the housekeeping part of the conversion? What I have in mind is a function that traverses the nested struct and calls a conversion function when a “leaf type” is encountered and returns a nested Dict). And, of course, a complementary function that traverses a nested Dict and copies the leaf types back into a nested struct.

I would even be willing to ensure that what I save is all mutable. And I could provide an initialized object that loaded values are copied into. That would cover probably 90% of my use cases.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 13, 2020, 12:15pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/11 "2020-07-13T12:15:30Z")

</div>

Maybe you missed it above, but I think the right approach is StructTypes.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [July 13, 2020, 12:32pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/12 "2020-07-13T12:32:46Z")

</div>

Maybe I am misreading the StructTypes docs, but this doesn’t seem to handle the housekeeping part (recursion through the nested structs).

An example of StructTypes being used to serialize / deserialize a nested struct would be helpful.

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [July 13, 2020, 12:45pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/13 "2020-07-13T12:45:25Z")

</div>

I’ve been using JLD2 for a long time are rarely had any issues with it, it would certainly good to keep it up-to-date, even with the current functionalities only.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 13, 2020, 12:45pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/14 "2020-07-13T12:45:53Z")

</div>

> [@hendri54](#):
>
> Maybe I am misreading the StructTypes docs, but this doesn’t seem to handle the housekeeping part (recursion through the nested structs).

No, you ned to define the method for all the types of the fields of all data structures involved. If the data type happens to be in another package you need to either do type piracy (not recommended) or convince the maintainer of the other package to depend on StructTypes to add the method (good luck)

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [July 13, 2020, 12:55pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/15 "2020-07-13T12:55:20Z")

</div>

That’s one reason why I was hoping for a deserializer that populates the fields of an already constructed object.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 13, 2020, 1:03pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/16 "2020-07-13T13:03:51Z")

</div>

Working with nested dictionaries is also not very fast.  
JLD2 is great in that it produces an inline representation for immutable structs.  
When you have lots of structs this can be much faster because of less indirection.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [July 13, 2020, 1:08pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/17 "2020-07-13T13:08:12Z")

</div>

> [@JonasIsensee](#):
>
> In my applications text-based formats such as `CSV` or `JSON` are not an option due to file-size  
> and because I don’t want rounding errors just from saving and loading my data.

If you want smallest file size, HDF5 supports compression (what JLD format is based on), but historically not SPDP (and thus I’m not sure JLD[2] does yet), while now possible for 30% more compression (but 10x faster is possible):

[https://userweb.cs.txstate.edu/~burtscher/research/SPDP/](https://userweb.cs.txstate.edu/~burtscher/research/SPDP/)

> SPDP is a fast, lossless, unified compression/decompression filter for HDF5 that has been designed for both 32-bit single-precision (float) and 64-bit double-precision (double) floating-point data. It also works on other data.

The paper on it is really interesting: [https://userweb.cs.txstate.edu/~mb92/papers/dcc18.pdf](https://userweb.cs.txstate.edu/~mb92/papers/dcc18.pdf)

> Abstract: Scientific computing produces, transfers, and stores massive amounts of single-and double-precision floating-point data, making this a domain that can greatly benefit from data compression. To gain insight into what makes an effective lossless compression algorithm for such data, we generated over nine million algorithms and selected the one that yields the highest compression ratio on 26 datasets.  
> […]  
> We named the resulting algorithm SPDP, which is an abbreviation for “Single Precision Double Precision”. It is brand new […] Only Zstd performs better. On average, SPDP outperforms Blosc, bzip2, FastLZ, LZ4, LZO, and Snappy by at least 30% in terms of compression ratio. However, it tends to be slower.

We should consider supporting, using:  
[https://juliahub.com/ui/Packages/TurboPFor\_jll/4zXB1/0.0.1+0](https://juliahub.com/ui/Packages/TurboPFor_jll/4zXB1/0.0.1+0)

i.e. this package: [GitHub - powturbo/TurboPFor-Integer-Compression: Fastest Integer Compression](https://github.com/powturbo/TurboPFor-Integer-Compression)

and: [GitHub - powturbo/Turbo-Transpose: Transpose: SIMD Integer+Floating Point Compression Filter](https://github.com/powturbo/Turbo-Transpose) [The first part is important as e.g. the current fastest supercomputer ARM-based and coming Macs too.]

> **ALL** TurboTranspose functions now available under **64 bits ARMv8** including **NEON** SIMD. […]
> 
> - Dynamic CPU detection and **JIT scalar/sse/avx2** switching
> - 100% C (C++ headers), usage as simple as memcpy […]
> - more efficient, up to **10 times!** faster than [Bitshuffle](https://github.com/powturbo/Turbo-Transpose#bitshuffle)
> - better compression (w/ lz77) and  
> **10 times!** faster than one of the best floating-point compressors [SPDP](https://github.com/powturbo/Turbo-Transpose#spdp)
> - can compress/decompress (w/ lz77) better and faster than other domain specific floating point compressors  
> […]  
> eTp4Lzt = lossy compression with allowed error = 0.0001

See also: [https://arxiv.org/pdf/1503.00638.pdf](https://arxiv.org/pdf/1503.00638.pdf)

> a nearly lossless rounding step which compares the precision of the data to a generalized and calibration-independent form of the radiometer equation. This allows the precision of the data to be reduced in a way that has an insignificant impact on the data. The newly developed Bitshuffle lossless compression algorithm is subsequently applied

Also interesting: [GitHub - Ed-von-Schleck/shoco: shoco is a compressor for small text strings](https://github.com/Ed-von-Schleck/shoco)

> for very small strings, it will always be better than standard compressors.

---

<div class="post-metadata">

**Author:** ![jlperla](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlperla/32/34332_2.png) [@jlperla](https://discourse.julialang.org/u/jlperla)\
**Post date:** [July 13, 2020, 4:28pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/18 "2020-07-13T16:28:32Z")

</div>

> [@Tamas\_Papp](#):
>
> if stability is not required (ephemeral storage), just serialize with `Serialization` .

Does this work cross-platform? Having something stable for a given julia version across different OSs seems like a reasonable solution.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 13, 2020, 5:05pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/19 "2020-07-13T17:05:47Z")

</div>

It is not guaranteed to, but I think it _may_ work in practice if both are either 32 or 64 bit and the endianness matches. But I don’t think you want to rely on this, it could break any time without prior announcement.

In practice, I would just choose a file format that fits the problem. CSV, JSON/BSON, HDF5, Feather… so many to choose from, with various advantages for each.

---

<div class="post-metadata">

**Author:** ![williamfgc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/williamfgc/32/15445_2.png) [@williamfgc](https://discourse.julialang.org/u/williamfgc)\
**Post date:** [July 13, 2020, 11:47pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/20 "2020-07-13T23:47:42Z")

</div>

Would people be interested in Julia bindings to adios2? (Disclaimer: I’ve been a core contributor over the years). We had a request for Julia bindings a while ago, but didn’t take off. I can work on those if there is enough interest.

Info about adios2:  
open access paper: [Redirecting](https://doi.org/10.1016/j.softx.2020.100561)  
repo: [https://github.com/ornladios/ADIOS2](https://github.com/ornladios/ADIOS2)  
docs: [https://adios2.readthedocs.io](https://adios2.readthedocs.io)

[Next page](https://discourse.julialang.org/t/a-future-for-jld2/42958.md?page=2)