# Suggested formats for saving and serialization

**URL:** <https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230>\
**Category:** Data\
**Tags:** package, data\
**Created:** [April 15, 2017, 10:10am UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230 "2017-04-15T10:10:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 15, 2017, 10:10am UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/1 "2017-04-15T10:10:55Z")

</div>

I was wondering what the suggested formats for saving and serializing are these days. I am really confused about the landscape of DataTables, DataFrames, databases, etc. I was hoping to write a few functions for the DiffEq solution type to save to some common data formats, but am not sure what I should be targeting. It should be something that would work well with statistics, plotting, and machine learning libraries. Or maybe the approach is just to go generic: I know there are things like DataStreams which are “independent readers”, is there something reverse that I can target so the user can choose which data type they want out? Would that even be necessary? I am hoping someone could guide me in the right direction here. Thanks!

For completeness, I opened an issue on DifferentialEquations.jl related to this topic (and it shows how idea-less I am, except I have had requests for something of this nature):

[https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/151](https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/151)

Thanks in advance for any ideas.

Also, is there by any chance a form of serialization for types which hold functions? I know JLD hasn’t worked since v0.5, and am wondering if there’s anything along these lines.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [April 15, 2017, 11:32am UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/2 "2017-04-15T11:32:42Z")

</div>

> [@ChrisRackauckas](#):
>
> DiffEq solution type to save to some common data formats, but am not sure what I should be targeting. It should be something that would work well with statistics, plotting, and machine learning libraries.

csv and HDF5 are probably the most widespread open formats for numerical data.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [April 15, 2017, 12:29pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/3 "2017-04-15T12:29:52Z")

</div>

DataStreams are not just for reading, they are also for writing. I would suggest implementing a DataStream `Source` a let the user choose what output format they want to use (possibly with a default format if you want).

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 17, 2017, 1:49pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/4 "2017-04-17T13:49:21Z")

</div>

Are there any examples for how to setup a DataStream `Source` anywhere? Are there examples how how to take an arbitrary source and write it to a DataFrame?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [April 17, 2017, 4:04pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/5 "2017-04-17T16:04:00Z")

</div>

You wouldn’t deal with `DataFrame` at all, you would just implement the `Source` interface, and the code living in DataFrames would take care of creating the object. For an example, you can have a look at the DataFrames code implementing a `Source`: [https://github.com/JuliaStats/DataFrames.jl/pull/1174](https://github.com/JuliaStats/DataFrames.jl/pull/1174) CSV.jl is another possibly useful example.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 17, 2017, 4:10pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/6 "2017-04-17T16:10:17Z")

</div>

I know I don’t need to write that code. But I was wondering what the code looks like for generating a DataFrame from an arbitrary Source.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [April 17, 2017, 7:12pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/7 "2017-04-17T19:12:49Z")

</div>

I just put together a quick and dirty integration with [IterableTables](https://github.com/davidanthoff/IterableTables.jl) in this PR [https://github.com/davidanthoff/IterableTables.jl/pull/22](https://github.com/davidanthoff/IterableTables.jl/pull/22). With that you can easily convert a `DESolution` into any of the supported table sink types, e.g. things like `DataTable(sol)` will work to create a `DataTable` from a `DESolution` instance. Essentially you get support for all the sinks that are listed in the README, plus of course full [Query](https://github.com/davidanthoff/Query.jl) integration, i.e. one can easily run queries against a `DESolution` instance. You also get integration with `DataStreams` from that “for free”: you can use `IterableTables.get_datastreams_source(sol)` to create a `DataStreams.Source` from your solution (I’m still trying to figure out an easier way to handle that particular integration from a user point of view).

I’m currently just waiting that the package to be registered: [https://github.com/JuliaLang/METADATA.jl/pull/8878](https://github.com/JuliaLang/METADATA.jl/pull/8878). And then we would have to clean out that PR a bit more before I could merge it.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 17, 2017, 7:22pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/8 "2017-04-17T19:22:26Z")

</div>

That sounds great! The API looks very simple too. That looks like the solution I was needing. Thanks! I’ll comment on the PR

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [April 17, 2017, 7:23pm UTC](https://discourse.julialang.org/t/suggested-formats-for-saving-and-serialization/3230/9 "2017-04-17T19:23:29Z")

</div>

> [@ChrisRackauckas](#):
>
> The API looks very simple too.

Don’t say that, it will just discourage me from writing the documentation I really should be writing for this 😉