# Save and restore DataFrame, and serialize()/deserialize()

**URL:** <https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699>\
**Category:** General Usage\
**Created:** [March 13, 2018, 7:40pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699 "2018-03-13T19:40:54Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [March 13, 2018, 7:40pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/1 "2018-03-13T19:40:55Z")

</div>

dear j wizards—is there a mechanism to save and restore a DataFrame in binary format to disk, with all type information? serialize() and deserialize() do not work for DataFrame in DataFrames.jl 0.11.5 .

incidentally, serialize() and deserialize() only have an iostream but not a filename based interface. would be nice to be built in and easy to implement.

regards, /iaw

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 13, 2018, 7:49pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/2 "2018-03-13T19:49:45Z")

</div>

[https://github.com/simonster/JLD2.jl](https://github.com/simonster/JLD2.jl)

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 13, 2018, 7:51pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/3 "2018-03-13T19:51:04Z")

</div>

Also worth trying: [https://github.com/MikeInnes/BSON.jl](https://github.com/MikeInnes/BSON.jl)

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 13, 2018, 7:51pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/4 "2018-03-13T19:51:26Z")

</div>

`JLD2` seems to work:

```julia
julia> using DataFrames

julia> df = DataFrame(A = [1,2,3], B=[1.0, 2.0, 3.0], C=["1", "2", "3"])
3×3 DataFrames.DataFrame
│ Row │ A │ B │ C │
├─────┼───┼─────┼─────┤
│ 1 │ 1 │ 1.0 │ "1" │
│ 2 │ 2 │ 2.0 │ "2" │
│ 3 │ 3 │ 3.0 │ "3" │

```

Save:

```julia
julia> using JLD2

julia> jldopen("df.jld2", "w") do file
         file["df"] = df
       end;

```

Load:

```julia
julia> jldopen("df.jld2") do file
           file["df"]
       end
3×3 DataFrames.DataFrame
│ Row │ A │ B │ C │
├─────┼───┼─────┼─────┤
│ 1 │ 1 │ 1.0 │ "1" │
│ 2 │ 2 │ 2.0 │ "2" │
│ 3 │ 3 │ 3.0 │ "3" │
```

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [March 13, 2018, 8:03pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/5 "2018-03-13T20:03:47Z")

</div>

[Feather](https://github.com/JuliaData/Feather.jl) is also quite nice, and is a standard format that is also available in other languages. I have a PR to it which drastically speeds it up and offers some really nice random access features. I think it is having to wait on some other things in the ecosystem to mature a little, but I hope to get it merged at some point in the not so distant future. The existing master should work fine though.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [March 13, 2018, 10:27pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/6 "2018-03-13T22:27:55Z")

</div>

thanks, everyone. I think the \*SON formats lose type information.

JLD2 looks good. alas, is this specialized and possibly slower overkill? (it seems designed as an interchange format, rather than as a “native” representation.)

Would it not be desirable/easy(?) to have a standard function, like serialize (or extend serialize), that is the generic equivalent of copy() (or deepcopy()), that would binary-write the full internal julia object representation to disk rather than to another memory location? (Could it then be read again?) . I do not know the julia internals, so this may be an absurd and ignorant suggestion.

related question: Is it possible to read back and evaluate the output of `dump()`?

regards,

/iaw

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 13, 2018, 11:22pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/7 "2018-03-13T23:22:19Z")

</div>

Isn’t that exactly what `serialize` already does?

```julia
julia> serialize(open("my_data", "w"), df)

julia> deserialize(open("my_data"))
3×3 DataFrames.DataFrame
│ Row │ A │ B │ C │
├─────┼───┼─────┼─────┤
│ 1 │ 1 │ 1.0 │ "1" │
│ 2 │ 2 │ 2.0 │ "2" │
│ 3 │ 3 │ 3.0 │ "3" │

```

The disadvantage of `serialize` is that it’s doing exactly as you suggest and writing a fairly literal dump of Julia’s model of an object to disk. That means that loading that file on a different Julia version is likely to be impossible.

JLD2, JSON, BSON, Feather, etc. all use widely-supported formats so that it’s more likely you can read your data in the future or in another language. As for performance, despite the fact that it’s not as literal as `serialize`, JLD2 is quite fast, and Simon showed in his JuliaCon 2017 presentation that JLD2 can even be faster than Julia’s built-in serialize in some cases: [JuliaCon 2017 | JLD2: High-Performance Serialization | Simon Kornblith - YouTube](https://www.youtube.com/watch?v=qc0dw-cAmLY)

I would suggest trying out one of the approaches and measuring the performance for yourself. As for reading the output of `dump()`,you could write one, but I would not recommend it. There’s no guarantee that `dump()` prints something that you could eval (it generally doesn’t) and there’s no guarantee that type you’re using has a constructor that matches the way data is printed by `dump()`. It really doesn’t seem worthwhile.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [March 13, 2018, 11:40pm UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/8 "2018-03-13T23:40:58Z")

</div>

thx, rdeits. embarrassingly, I had omitted the first argument to serialize, and then misinterpreted the resulting error as serialize not being able to work on dataframes.

I was also surprised about your sticking the open into the first argument. interesting. does it flush and close the file at IOStream() destruction time?

regards,

/iaw

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 14, 2018, 12:12am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/9 "2018-03-14T00:12:47Z")

</div>

> does it flush and close the file at IOStream() destruction time?

No, the file may not be closed until Julia decides to finalize it. For a more robust version, you can use the `open() do...end` construction:

```julia
julia> open("my_data", "w") do file
         serialize(file, df)
       end

```

which does guarantee that the file will be flushed and closed at the end of the `do...end` block. This is in the help for `?open`:

```julia
open(f::Function, command, mode::AbstractString="r", stdio=DevNull)

  Similar to open(command, mode, stdio), but calls f(stream) on the resulting read or write stream, then closes the
  stream and waits for the process to complete. Returns the value returned by f.

```

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [March 14, 2018, 12:30am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/10 "2018-03-14T00:30:42Z")

</div>

pity. the do end construction is a little more verbose. ☹

regards,

/iaw

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 14, 2018, 12:33am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/11 "2018-03-14T00:33:05Z")

</div>

True, but you can write the same thing as a one-liner if you want:

```julia
open(file -> serialize(file, df), "my_data", "w")

```

and keep the guarantees that the file will be immediately closed.

There’s also a proposed shorthand anonymous function syntax that _might_ make it into v1.0 or v1.1 that would make `serialize(_, df)` equivalent to `file -> serialize(file, df)`, so you could do:

```julia
open(serialize(_, df), "my_data", "w")

```

which is pretty good, I think 😄

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [March 14, 2018, 12:40am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/12 "2018-03-14T00:40:33Z")

</div>

ah, I see. this is what the `open(f::Function` was for. something that had not clicked. (julia innovation, I think.)

and the \_ anonymous notation is even nicer.

/iaw

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 14, 2018, 9:03am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/13 "2018-03-14T09:03:17Z")

</div>

> [@iwelch](#):
>
> JLD2 looks good. alas, is this specialized and possibly slower overkill?

With Julia moving so rapidly, the serialization format changes accordingly. If you dig up these files in a few months, your future self will thank you for choosing JLD2 (or a similar format, like BSON).

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [September 13, 2019, 5:00am UTC](https://discourse.julialang.org/t/save-and-restore-dataframe-and-serialize-deserialize/9699/14 "2019-09-13T05:00:41Z")

</div>

Has serialization format stabilised now that Julia 1.1 is out? I note the `Serialization.serialize` function is noted as from Julia 1.1 onwards.