# What is the preferred way to save variables?
**URL:** https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918
**Category:** General Usage
**Tags:** jld, hdf5, jld2
**Created:** [March 15, 2019, 4:36pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918 "2019-03-15T16:36:11Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@zaen](https://discourse.julialang.org/u/zaen)
#### Post date: [March 15, 2019, 4:36pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/1 "2019-03-15T16:36:11Z")
I’m trying to save variables from one run of a problem JuMP to use as initial values for variables in a separate JuMP problem. I see that there seem to be several (competing?) options for doing this:
- [JLD](https://github.com/JuliaIO/JLD.jl)
- [JLD2](https://github.com/JuliaIO/JLD2.jl)
- [BSON](https://github.com/MikeInnes/BSON.jl)
- [HDF5](https://github.com/JuliaIO/HDF5.jl)
- [MAT](https://github.com/JuliaIO/MAT.jl)
- [CSV](https://github.com/JuliaData/CSV.jl)
- [JSON](https://github.com/JuliaIO/JSON.jl)
JLD2’s readme says it’s the successor to JLD, but over the past several months JLD has been continuously updated while JLD2 has not. However, looking through the issues on the JLD repository, I came across [an issue](https://github.com/JuliaIO/JLD.jl/issues/243) where one of the maintainers said that JLD would only be getting compatibility fixes rather than updates in the future.
From [another post on the forum](https://discourse.julialang.org/t/how-to-save-variables-from-the-workspace/11710) I found BSON, which seems to be completely independent of the JLD variants.
There’s an HDF5 package as well, but in its readme it talks about the advantages of using JLD over pure HDF5.
I don’t know how well the last three packages handle Julia data types, but my guess is that they’ll work for basic things like arrays and strings but have trouble with more complex types.
Which of these, if any, is the preferred or “official” way to save workspace variables?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [March 15, 2019, 5:07pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/2 "2019-03-15T17:07:47Z")
> [@zaen](#):
>
> Which of these, if any, is the preferred or “official” way to save workspace variables?
There are four, somewhat but not completely orthogonal questions here:
1. well-defined formats with a spec and more ad-hoc solutions (eg HDF5 vs CSV),
2. binary vs text-based formats (eg BSON vs JSON), which can result in loss of precision in some cases,
3. saving a restricted set of values (eg strings, arrays, numbers) or generic Julia values (array of some `struct`s, further complications),
4. maintenance status.
No solution is perfect, and you need to be more precise about your requirements.
FWIW, I think that while saving and reloading native objects would be very nice, in practice it is rather brittle; and most existing packages have outstanding issues, often with data corruption, so that rules out JLD2 and BSON. JLD, as you said, is not intensively maintained either. HDF5 is promising on paper but it is a monolithic, single-implementation piece of software (everyone just wraps the C library, if you ignore JLD2, which implements a subset). CSV is not precisely defined but can work in practice for matrix-like data, while JSON and BSON can be more flexible, but still require that you reformat your data to what they support.
In practice, I usually go with CSV, or Serialization for short-term storage on the same machine. This is overly conservative, but in the past I was burned by HDF5, JLD2, and BSON (corrupted data or intermittent bugs that were not fixed for a while). With CSV, I can always rescue my data, even if I have to whip up a parser myself in the worst case.
There are various other fringe formats, but they are even less widely used.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [March 15, 2019, 5:27pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/3 "2019-03-15T17:27:59Z")
I’m still in the early stages, so perhaps I should be careful about advertising it in a public forum like this, but I’m currently very much in the process of completing my pure Julia implementation of the [apache arrow](https://arrow.apache.org/) standard. This is essentially a table-focused format, however the full standard is diverse enough that it allows for storing a wide variety of objects, in principle it should support storing a wide class of Julia `struct`s as tuples. The advantage of using arrow is that it is supported in a wide variety of languages, and it is useful for interacting with things like Parquet and Spark. (And yes, they don’t advertise it well on the website but it can certainly be used for storing data on disk.)
I am constantly needing to stash what I am working on on disk or on remote storage such as S3, and the time may well come when I need a Julia Spark wrapper on par with pyspark, so I’ve had lots of motivation for working on arrow.
On the opposite end of the compatibility spectrum, note that Julia’s [built-in serializer](https://docs.julialang.org/en/v1/stdlib/Serialization/) can be exceptionally simple to use, was very performant the last time I checked, and is very reliable. The big disadvantage is of course that compatibility between different Julia versions is not guaranteed, so it’s only suitable for very short-term use.
---
### Author:  [@mauro3](https://discourse.julialang.org/u/mauro3)
#### Post date: [March 15, 2019, 5:59pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/4 "2019-03-15T17:59:36Z")
One more [https://github.com/eschnett/ASDF.jl](https://github.com/eschnett/ASDF.jl)
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [March 15, 2019, 6:03pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/5 "2019-03-15T18:03:33Z")
I really want to use ASDF just to annoy my data science colleagues.
Would be cool if it had a pure Julia implementation though 😉
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [March 16, 2019, 9:57pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/6 "2019-03-16T21:57:36Z")
[Protobuf](https://github.com/JuliaIO/ProtoBuf.jl) may be a good choice? Does anyone have experience with that?
---
### Author:  [@bennedich](https://discourse.julialang.org/u/bennedich)
#### Post date: [March 16, 2019, 10:28pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/7 "2019-03-16T22:28:42Z")
I haven’t used the Julia implementation, but my experience of Protobuf from other contexts is very good: compact storage and extremely fast deserialization speeds. However, if you don’t need fast deserialization, I would choose another format for more readability and compatibility, such as CSV or JSON.
---
### Author:  [@zaen](https://discourse.julialang.org/u/zaen)
#### Post date: [March 18, 2019, 4:32pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/8 "2019-03-18T16:32:15Z")
> [@Tamas\_Papp](#):
>
> There are four, somewhat but not completely orthogonal questions here:
>
> 1. well-defined formats with a spec and more ad-hoc solutions (eg HDF5 vs CSV),
> 2. binary vs text-based formats (eg BSON vs JSON), which can result in loss of precision in some cases,
> 3. saving a restricted set of values (eg strings, arrays, numbers) or generic Julia values (array of some `struct`s, further complications),
> 4. maintenance status.
>
> No solution is perfect, and you need to be more precise about your requirements.
Ideally, I’d like a format that is binary, can save generic Julia values, and is maintained, but it looks like there’s not an option that meets all of those requirements currently. Serialization is good, but as has been pointed out, is incompatible between Julia versions. I’ll probably stick to a combination of that and CSV for the time being. Good luck with your project, @ExpandingMan.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [March 18, 2019, 4:49pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/9 "2019-03-18T16:49:35Z")
How does ASDF compare to Arrow, feather, hdf5, parquet…?
A database that seems very performant is TileDB but we don’t have it on Julia yet.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [March 18, 2019, 4:49pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/10 "2019-03-18T16:49:49Z")
> [@zaen](#):
>
> . Good luck with your project, @ExpandingMan.
Thanks! But, to be clear, arrow will not allow you to “save generic Julia values”, at least not easily. Like I indicated, if you really wanted to you could, for example decompose some `struct`s as tuples and store those in an arrow format. One of the top things on my own personal wishlist is being able to store arrays of arbitrary rank, which the (I think new) arrow tensor format should accommodate. If it some point someone wanted to make a `JLDArrow` or something like that, that might be pretty cool, but I’m not likely to take that on myself.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [March 18, 2019, 4:51pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/11 "2019-03-18T16:51:51Z")
> [@zaen](#):
>
> can save generic Julia value
I think this is the hardest part, since Julia values can be really, really complex. Consider
```julia
struct Foo{T,S}
x::S
end
T = Union{Missing,Foo,Float64}
f = Foo{3,Vector{T}}(T[missing, missing, 1.0])
f.x[2] = f
```
The other complication is that types need to be defined before instantiating objects.
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [March 19, 2019, 12:33pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/12 "2019-03-19T12:33:33Z")
About the “saving arbitrary Julia variables” part of the request, I’m currently working on a library to handle this sort of thing without running into the maintenance burden of JLD or JLD2.
Specifically, my approach is a simple, lightweight package that allows IO packages like HDF5.jl or CSV.jl to expose “backends” to the user, that just do the very simple job of format-specific IO for the limited set of types that the format can natively support.
How more complicated types will be handled, is by creating a mapping function from the user’s types to some collection of types that the IO package natively supports, using metadata (potentially stored within the IO format, or outside of it, depending on what the IO package supports) to identify that that mapping looks like when the data is reloaded later on.
I’m hoping this generic approach will end up being less difficult to maintain than the JLD\* packages since it doesn’t directly have to implement any HDF5 specific details, and can make use of multiple backends to support saving/loading (almost) anything the user can throw at it.
Once I’ve finished up the first pass at functionality I’ll make a release announcement on Discourse (it’ll probably be called SerDes.jl); it might be in a few weeks since I’ve got a lot of other stuff going on in my life.
---
### Author:  [@mauro3](https://discourse.julialang.org/u/mauro3)
#### Post date: [March 19, 2019, 12:47pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/13 "2019-03-19T12:47:17Z")
Will it handle closures?
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [March 19, 2019, 1:13pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/14 "2019-03-19T13:13:32Z")
I’m not sure whether I’m hoping for a “yes” or a “no”…
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [March 19, 2019, 2:16pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/15 "2019-03-19T14:16:08Z")
We’ll give it a try, but it’s not a top priority. I’m not going to prevent any PRs to support closures or any other kind of code-mixed-with-data serdes, of course!
---
### Author:  [@Usor](https://discourse.julialang.org/u/Usor)
#### Post date: [November 2, 2019, 10:53am UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/16 "2019-11-02T10:53:50Z")
I saw some forum threads, and some about the current state of things. Am I right to think that I may as well forget about JLD, JLD2, HDF5 and most other such packages and ways for Julia?
Except for BSON.jl? What is its status? Should you consider it the safe way?
---
### Author:  [@FHell](https://discourse.julialang.org/u/FHell)
#### Post date: [November 2, 2019, 11:14am UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/17 "2019-11-02T11:14:00Z")
We have moved to BSON.jl and it seems to work fairly reliably for the time being. Haven’t stress tested it in cluster environments yet though.
---
### Author:  [@spaceLem](https://discourse.julialang.org/u/spaceLem)
#### Post date: [November 2, 2019, 12:20pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/18 "2019-11-02T12:20:27Z")
I have been using `JLD2` for my work, and it’s done the job pretty well, storing and retrieving a lot of hetergeneous data (csv is not suitable for my case). I’m interested in seeing what else people settle on using though, I’d rather go with the standard.
---
### Author:  [@Usor](https://discourse.julialang.org/u/Usor)
#### Post date: [November 2, 2019, 12:23pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/19 "2019-11-02T12:23:30Z")
You mean BSON.jl has become the standard option? In your reasonable estimation, do you think BSON.jl will then remain the go-to option?
---
### Author:  [@Usor](https://discourse.julialang.org/u/Usor)
#### Post date: [November 2, 2019, 12:25pm UTC](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/20 "2019-11-02T12:25:04Z")
I read about serious problems with JLD2. Their page also mentions instability.
[Next page](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918.md?page=2)