# \[ANN\] JDF.jl v0.2.0 - Julia DataFrames serialization format

**URL:** <https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217>\
**Category:** Package Announcements\
**Created:** [October 23, 2019, 12:18pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217 "2019-10-23T12:18:19Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2019, 12:18pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/1 "2019-10-23T12:18:19Z")

</div>

JDF is the Julia DataFrames serialization format. It’s a specialised serialization format and hence doesn’t support arbitrary objects like JLD2 and JLSO. This loss of generality is more than made up for in gains in speed and reliability for saving and loading DataFrames.

JDF now supports `DataFrames` containing these types

- `WeakRefStrings.StringVector`
- `Vector{T}`
- `CategoricalArrays.CategoricalVetors{T}`

where `T` can be `String`, `Bool`, and `isbits` types i.e. `UInt*`, `Int*`,  
and `Float*` `Date*` types etc.

`RLEVectors` support will be considered in the future when `missing` support  
arrives for `RLEVectors.jl`.

Also, there is now the ability to **load only the columns you select**. For example

`a2_selected = loadjdf("iris.jdf", cols = [:species, :sepalLength, :petalWidth])`

From JDF.jl v0.2, I am committed to making **all JDF files loadable in ALL future version of JDF.jl**.

Please see Github

> **[GitHub - xiaodaigh/JDF.jl: Julia DataFrames serialization format](https://github.com/xiaodaigh/JDF.jl)**
>
> Julia DataFrames serialization format. Contribute to xiaodaigh/JDF.jl development by creating an account on GitHub.

If you find JDF.jl useful, please do Star the github repo. It keeps me going 🙂

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [October 23, 2019, 12:32pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/2 "2019-10-23T12:32:46Z")

</div>

This is interesting for us. We use `DataFrame` in DrWatson’s collect results functionality: [Running & Listing Simulations · DrWatson](https://juliadynamics.github.io/DrWatson.jl/dev/run&list/#Collecting-Results-1) . What we do is we scan your directory and make all your simulations a `DataFrame` and then save it. We re-use existing dataframes, which also can get big, so performance gains in terms of read and write are important.

At the moment though I don’t think we can move into JDF, because of the type limitations. I’ll keep watching this post to see if there are less restrictions on types as time progresses. ( @JonasIsensee I’m tagging you this may be interesting for you )

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2019, 12:34pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/3 "2019-10-23T12:34:57Z")

</div>

What types do you need? If it’s a small list I can try to prioritise them.

In JDF.jl v0.3 (the next version) there won’t be type restrictions, but some types might be slow to safe though as there may not be specialised algorithms for saving them.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [October 23, 2019, 12:40pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/4 "2019-10-23T12:40:52Z")

</div>

Yeah, that’s the problem: I don’t know in advance what types users may have created in their simulations that they want to save. Seems like 0.3 does exactly what we need though!

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [October 23, 2019, 12:47pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/5 "2019-10-23T12:47:45Z")

</div>

> [@xiaodai](#):
>
> `String`

By the way, I think `Symbol` should also have a “fast” implementation. AT the moment I use Symbols as parameters to represent complicated functions that I don’t want to save in my DataFrame, and during my simulations I `@eval` those symbols.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2019, 12:57pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/6 "2019-10-23T12:57:12Z")

</div>

> [@Datseris](#):
>
> I don’t know in advance what types users may have created in their simulations that they want to save.

In that case, JDF.jl may only yield speeed benefit if you use Julia 1.3 because of multithreading, because JDF would need to rely on JLSO.jl or the like for serialization arbitray format anyway.

> [@Datseris](#):
>
> think `Symbol` should also have a “fast” implementation

Funny that! I was just thinking about `Symbol`s. JDF.jl v0.2.1 it is!

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [October 23, 2019, 1:01pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/7 "2019-10-23T13:01:48Z")

</div>

Hey, this is cool work!

I have one comment on the types:  
At the moment we have columns with custom types (or collections of types - \> Array of Any)  
in our aggregated DataFrames but i suppose this is not really necessary if we find a different proper representation.  
Strings are probably not effective as you can’t query into the values easily anymore.  
Converting to namestuples could be an option or would that also not work fast?

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2019, 1:07pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/8 "2019-10-23T13:07:03Z")

</div>

You can save NamedTuple already provided that all the variable inside the NamedTuple is `isbits` and are **of the same structure**. See example:

```julia
using DataFrames

adf = DataFrame(a = [(ok = 2, lah = 2), (ok = 3, lah = 3)])

savejdf(adf, "c:/plsdel.jdf")
adf_copy = loadjdf("c:/plsdel.jdf") # same as adf

adf_copy == adf # true

```

and it’s pretty fast and that’s because

```julia
isbits((ok = 2, lah = 2)) # is true

```

But you have to make sure that every element in your `Vector{NameTuple}` has the same `NameTuple` structure. Or it will fail, which I need better error messages for.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2019, 2:00pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/9 "2019-10-23T14:00:15Z")

</div>

> [@Datseris](#):
>
> `Symbol`

Just tagged JDF.jl 0.2.1 with `Symbol` support. Doesn’t satisfying your use-case yet. But it’s something

See [New version: JDF v0.2.1 by JuliaRegistrator · Pull Request #4657 · JuliaRegistries/General · GitHub](https://github.com/JuliaRegistries/General/pull/4657)

---

<div class="post-metadata">

**Author:** ![mwsohn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mwsohn/32/2696_2.png) [@mwsohn](https://discourse.julialang.org/u/mwsohn)\
**Post date:** [October 25, 2019, 12:09am UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/10 "2019-10-25T00:09:46Z")

</div>

This is great. Can it handle Dates and DateTimes?

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 25, 2019, 12:22am UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/11 "2019-10-25T00:22:36Z")

</div>

Yes. See example

```juila
using DataFrames, JDF, Dates

df = DataFrame(d = DateTime.(2013:2014), d1 = Date.(2013:2014))

savejdf(df, "date.jdf")

loadjdf("date.jdf")

```

> [@xiaodai](#):
>
> `Date*` types etc.

In fact, all `isbits` type are supported. Also `structs` whose elements are all `isbits` types are supported as well. `TimeZones.jl` support is coming in an upcoming release, too.

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [May 19, 2020, 8:38pm UTC](https://discourse.julialang.org/t/ann-jdf-jl-v0-2-0-julia-dataframes-serialization-format/30217/12 "2020-05-19T20:38:14Z")

</div>

I just tried it out - very fast and small file sizes.  
Great, thanks for your work!