# A future for JLD2?

**URL:** <https://discourse.julialang.org/t/a-future-for-jld2/42958>\
**Category:** Community\
**Tags:** jld2\
**Created:** [July 12, 2020, 7:05pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958 "2020-07-12T19:05:00Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2020, 12:24am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/21 "2020-07-14T00:24:57Z")

</div>

> [@williamfgc](#):
>
> Would people be interested in Julia bindings to adios2?

I’d say that bindings for good libraries in other languages are always welcome, but this is a bit tangential to the current discussion, where many users advocate for a good, stable pure-Julia solution for serialisation.

---

<div class="post-metadata">

**Author:** ![tisztamo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tisztamo/32/16200_2.png) [@tisztamo](https://discourse.julialang.org/u/tisztamo)\
**Post date:** [July 14, 2020, 5:11am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/22 "2020-07-14T05:11:22Z")

</div>

I have tried to use Serialization with small but relatively complex, parametric structs between an Ubuntu 18.04 and a recent OSX. It worked on Julia 1.3 but upgrading both to 1.4 broke it.

Edit: IMO stabilizing Serialization would be a great solution to this problem. I would also be happy to help doing it.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [July 14, 2020, 6:39am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/23 "2020-07-14T06:39:07Z")

</div>

> [@lhupe](#):
>
> There are applications where `NaN` (i.e. the result is known to not exist) and `missing` (i.e. we don’t know the result) have to be distinguished.

`NaN`s can carry a “payload”. That is: there are a few million different 32-bit values that are all interpreted as `NaN`. (Even more for 64-bit numbers.)

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 14, 2020, 7:26am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/24 "2020-07-14T07:26:03Z")

</div>

Thank you, these are all interesting discussions  
but by now the conversation has drifted away significantly from what I wanted to talk about in this thread.

To restate the question:  
Are people already using JLD2 or would use it if it got maintained again?  
Are you able and willing to help?  
I would also particularly ask for feedback from JuliaIO people.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [July 14, 2020, 11:09am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/25 "2020-07-14T11:09:08Z")

</div>

Thanks for working on JLD2. If it’s actually still faster (or can be), then it ~~seems~~ might be worthwhile. I’m just not sure it is faster than JLD when non-default Blosc compression is used by it.

I only have modest use cases (so far), so I don’t care either way, but I looked a bit into both. I noticed JLD2 gets me the same file independently of compression option:

```julia
julia> A = []
julia> file = jldopen("mydata_c.jld2", "w", compress=true)
julia> file["A"] = A # for JLD I could do (not without JLD. as in docs so I filed an issue): JLD.@write file A
julia> close(file)

```

For JLD the files differ, while same size (I guess couldn’t for non-empty array):

```julia
-rw-r--r-- 1 pharaldsson_sym pharaldsson_sym 6288 júl 14 10:05 mydata.jld
-rw-r--r-- 1 pharaldsson_sym pharaldsson_sym 6288 júl 14 10:52 mydata_c.jld
-rw-r--r-- 1 pharaldsson_sym pharaldsson_sym 5180 júl 14 10:49 mydata.jld2
-rw-r--r-- 1 pharaldsson_sym pharaldsson_sym 5180 júl 14 10:33 mydata_c.jld2

```

Reading the code it seems JLD2 only has, and I guess, always uses deflate/ZlibCompressor, while JLD:

> uses [Blosc](http://www.blosc.org/) compression, which imposes very little performance penalty, but leads to HDF5 files that are not readable by other applications unless a Blosc plugin is installed. If you also specify `compatible=true` , then a different (and often slower) compression method is used that should be readable by any HDF5-using software.

I would like even better compression options that are now available, but with these used, I’m just not sure JLD2 is faster (is that outdated info, or just with memory mapped files or?).  
JLD, or HDF5.jl that is, added Mmap in July 2019: [Update to use BinaryProvider by staticfloat · Pull Request #555 · JuliaIO/HDF5.jl · GitHub](https://github.com/JuliaIO/HDF5.jl/pull/555/files)

and in Blosc in August 2018: [https://github.com/JuliaIO/HDF5.jl/blob/c7865be2431406f9438056580cbb891e375bfafb/Project.toml](https://github.com/JuliaIO/HDF5.jl/blob/c7865be2431406f9438056580cbb891e375bfafb/Project.toml)

From my perspective it’s time better spent on that, and (fully) Julia-only solutions not important, or preferred. It least the good compression algorithms are not in Julia, and will not be maintained there, in the near-future, the new ones, and also Blosc.

I’m not sure this applies to JLD2 however, and it’s a bit alarming for JLD:

> **Note:** You should only read JLD files from trusted sources, as JLD files are capable of executing arbitrary code when read in.

JLD2 has faster startup (despite HDF5\_jll now used by JLD):

```julia
$ julia --startup-file=no -O0 -q

julia> @time using JLD
  1.270251 seconds (1.66 M allocations: 94.025 MiB, 0.98% gc time)

julia> @time using JLD # default Julia optimization:
  1.632788 seconds (1.66 M allocations: 93.993 MiB, 0.87% gc time)

```

FYI: e.g. “JLD.jl is the preferred way of saving ScikitLearn.jl models.” because of “PyCallJLD, which uses pickle” [Links don't work and docs outdated · Issue #85 · cstjean/ScikitLearn.jl · GitHub](https://github.com/cstjean/ScikitLearn.jl/issues/85#issuecomment-653262703) [GitHub - JuliaPy/PyCallJLD.jl: JLD support for PyCall objects](https://github.com/JuliaPy/PyCallJLD.jl)

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 14, 2020, 12:22pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/26 "2020-07-14T12:22:33Z")

</div>

Another question that needs to be resolved is whether members of JuliaIO are willing to review my PRs.  
If that is not the case then I could still start developing my own fork of JLD2 and possibly register it at a later point in time.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 14, 2020, 2:21pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/27 "2020-07-14T14:21:29Z")

</div>

> [@JonasIsensee](#):
>
> whether members of JuliaIO are willing to review my PRs

I can’t speak for them, but I have seen (semi-)abandonned packages just add new contributors when someone wants to continue the work.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2020, 3:01pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/28 "2020-07-14T15:01:31Z")

</div>

The membership in the organisation is the last problem, you only need to find anyone with good understanding of the codebase to review the changes.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 14, 2020, 4:28pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/29 "2020-07-14T16:28:35Z")

</div>

That is probably true.  
The only alternative would be for me to start building a comprehensive summary of the library internals to make review easier and to keep development in a separate dev branch that needs to be tested rigorously before anything is merged into master.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 14, 2020, 4:45pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/30 "2020-07-14T16:45:04Z")

</div>

Hey guys, I’ve been following this discussion. Do you know of _any_ programming language where this “persistent” programming with arbitrary data structures works? That is what JLD2 is (was?) trying to do, isn’t it? Just run the program, at an arbitrary point dump the state of the program, and it can be resumed precisely as it was at any time.

Edit: I think it works with lisp, but is there anything else?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [July 14, 2020, 5:00pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/31 "2020-07-14T17:00:21Z")

</div>

ah yes…[https://root.cern.ch](https://root.cern.ch), (which is C++ but complex enough it’s basically its own thing);  
for example, [Chapter: Trees](https://root.cern.ch/root/htmldoc/guides/users-guide/Trees.html#example-2-a-tree-with-a-c-structure)

if anyone wants to actually use it, use python (official) binding: [ROOT: PyRoot tutorials](https://root.cern.ch/doc/master/group __tutorial__ pyroot.html)

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [July 14, 2020, 5:29pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/32 "2020-07-14T17:29:47Z")

</div>

I come from [Matlab](https://www.mathworks.com/help/matlab/matlab_oop/understanding-the-save-and-load-process.html) where this has worked for a long time.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 14, 2020, 5:37pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/33 "2020-07-14T17:37:32Z")

</div>

I haven’t used it recently, so I don’t know: has saving code worked at any point? Meaning if I define a function and try to save it, will it work?

---

<div class="post-metadata">

**Author:** ![Paulo\_Jabardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulo_jabardo/32/3196_2.png) [@Paulo\_Jabardo](https://discourse.julialang.org/u/Paulo_Jabardo)\
**Post date:** [July 14, 2020, 5:40pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/34 "2020-07-14T17:40:45Z")

</div>

The function `save` in R works really well, saving data and code.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 14, 2020, 6:30pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/35 "2020-07-14T18:30:22Z")

</div>

Root is impressive! But, it is still C++ ☹

---

<div class="post-metadata">

**Author:** ![jondeuce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jondeuce/32/16378_2.png) [@jondeuce](https://discourse.julialang.org/u/jondeuce)\
**Post date:** [July 14, 2020, 6:48pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/36 "2020-07-14T18:48:11Z")

</div>

Yes, you can save and load closures, user-defined classes, etc. Of course, this is considerably easier to do in Matlab than Julia, but it has been possible for quite some time.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [July 14, 2020, 6:50pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/37 "2020-07-14T18:50:37Z")

</div>

I’ll just add that I would use something like JLD2 if it was a bit more polished and well-tested. In the past, I did use MATLAB’s `save` utility quite extensively, even for longer term storage. Right now in Julia, I stick to CSV files and am just pickier about what actually needs to be saved to a file.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [July 14, 2020, 8:51pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/38 "2020-07-14T20:51:31Z")

</div>

I would like to help out (pending my time commitments elsewhere). One thing that I think could be useful is splitting out the HDF5 part from the serializer: I think there would be great value in having a pure-Julia HDF5 implementation, and potentially supporting other container file formats.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [July 15, 2020, 9:52am UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/39 "2020-07-15T09:52:15Z")

</div>

Thank you, that is great to hear!

> One thing that I think could be useful is splitting out the HDF5 part from the serializer: I think there would be great value in having a pure-Julia HDF5 implementation,

In principle I would agree but I fear that JLD2 is as fast as it is largely because it’s routines are fairly lowlevel. I’m not sure that this can be split off easily or without having to make a lot of compromises.  
On the other hand, I think it could be possible to add a HDF5 compatible mode  
that only supports basic types and encodes type signatures in a way that can be read by other HDF5 libraries.

This is definitely some that we can discuss in the future.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 15, 2020, 2:56pm UTC](https://discourse.julialang.org/t/a-future-for-jld2/42958/40 "2020-07-15T14:56:27Z")

</div>

HDF5 will maintain a significant presence in HPC due to its inclusion in the software for exascale project ([e4s-project.github.io](http://e4s-project.github.io)). That is something to be considered in this context, perhaps?

[Previous page](https://discourse.julialang.org/t/a-future-for-jld2/42958.md?page=1)

[Next page](https://discourse.julialang.org/t/a-future-for-jld2/42958.md?page=3)