# Suggestion for sometimes faster data serialization

**URL:** <https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302>\
**Category:** Internals & Design\
**Created:** [December 15, 2019, 1:28pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302 "2019-12-15T13:28:52Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 15, 2019, 1:28pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/1 "2019-12-15T13:28:52Z")

</div>

Hey Julians,

So I had this idea, and I was going to make an issue on a requisite package for it, then I realized the context of it is potentially broader then any single package.

So in the geospatial domain a trick for reducing file size is to cleave off floating point noise from the digits that are significant in all stored files (well known trick). Now, in many other domains this is also true! Some datastores only need 4-8 byte precision, but after say manipulation they end up with all this floating point junk.

Is there a foreseeable benefit for a widespread pattern to be implemented in say CSV.jl, JSON.jl, XLSX.jl, JuliaDB.jl, parquet.jl, etc for allowing a “Significant digit” truncation? I realize this is one of those semidangerous things, if an end user “thinks they know” the truncation level and messes up well that sucks for them, but if you genuinely know, it’s kind of a godsend for large swaths of I/O.

I could imagine some type wrappers for some of these columnar I/O libraries specifying the significant digits for each column to save on bytes. Maybe some formats are already handling this to some extent?

Anyways, just a passing thought I had while trying to optimize my own ZMQ streams…

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 15, 2019, 1:32pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/2 "2019-12-15T13:32:07Z")

</div>

Can’t you just transform to `Float32` or similar, and serialize that?

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 15, 2019, 1:34pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/3 "2019-12-15T13:34:59Z")

</div>

That’s one possibility definitely. Maybe this is a moot point actually. I don’t know the guts of how JSON/CSV reads plain text and turns it into native types. I suppose it probably isn’t doing anything at the byte level so probably no performance gains. Then, for binary file stores, I suppose skipping the first N bytes or trailing N bytes isn’t a huge gain either. Maybe someone with more knowledge of the internals knows better that this is pointless?

For parsing string types this nice though. Less char reads, smaller disk & memory footprint.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [December 15, 2019, 2:17pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/4 "2019-12-15T14:17:36Z")

</div>

I think it could help reduce file size for plain text formats (such as JSON). Maybe one could just convert (or round) to a fixed point type like [https://github.com/JuliaMath/FixedPointDecimals.jl](https://github.com/JuliaMath/FixedPointDecimals.jl) and then write it to a file from there?

I have some rotatable 3D plots at the bottom of this post [Locally maximizing the Rényi entropies | EPH](https://ericphanson.com/blog/2018/locally-maximizing-the-renyi-entropies/) that I made a couple years ago with Plots.jl and the plotlyjs backend (one of my first experiments with Julia!) and when that page loads, the browser downloads some (plaintext) .js files with hardcoded data points each specified to 16 digits, which seems pretty wasteful. I’m sure there’s a better way by writing some JavaScript to load the data from a binary format, but this was the easiest— and anyway, I just wanted to show a collaborator some 3D plots. I think it would cut down on file size without affecting the plot quality just by truncating to 5 or 6 digits. So to my mind there’s definitely a use-case for the technique.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 15, 2019, 2:22pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/5 "2019-12-15T14:22:56Z")

</div>

Beautiful plots! For the case of 2-D plots, we could easily determine pixel resolution and use that as a cut off. 3-D would be harder. I can foresee a use case here for some file types and applications. I’m just not exposed to the internals of all the IO libraries like some people are. Nor am I computer scientist and never tried to play one on TV.

Hmmm… Fixed point decimals could be handy for this.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 15, 2019, 4:58pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/6 "2019-12-15T16:58:58Z")

</div>

If you care about efficiency that much you should maybe be using binary formats, not text formats like CSV or JSON. Of course, transformating to a lower precision for storage is often a good option for binary formats as well.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [December 15, 2019, 5:53pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/7 "2019-12-15T17:53:00Z")

</div>

The [laszip](https://laszip.org/) compression uses this principle to attain impressive compression ratios.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 15, 2019, 6:32pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/8 "2019-12-15T18:32:15Z")

</div>

@stevengj - I agree with that, and that’s kind of where I am looking once I find a nice route to do that with. Maybe BSON for me personally, but we’ll see. But, for some people, CSV’s and JSON are nice because they don’t require any special or seasonal libraries to open them up and see whats inside. Hundreds of thousands of people use those files daily to perform rudimentary analysis’ and it’s a drain on the world to store noise/null values. The main niche I guess is someone boss could open them in excel, or notepad and see something, and corrupting plain text is pretty difficult. For archival purposes that’s kinda nice, although I think many would disagree that a storage format should be based on ease of technical access.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 15, 2019, 9:17pm UTC](https://discourse.julialang.org/t/suggestion-for-sometimes-faster-data-serialization/32302/9 "2019-12-15T21:17:08Z")

</div>

Try `JDF.type_compress!(df, compress_float=true) `

[https://github.com/xiaodaigh/JDF.jl](https://github.com/xiaodaigh/JDF.jl)