# The poor state of fileformats for High Performance computing
**URL:** https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326
**Category:** General Usage
**Created:** [August 11, 2017, 7:57am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326 "2017-08-11T07:57:53Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [August 11, 2017, 7:57am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/1 "2017-08-11T07:57:53Z")
About two weeks ago I have put a post asking for recommending the file-format use with high-perfomance computing. Our use-case is training of relatively simple neural networks but using terabytes of data. We do that on a single machine with GPU, where there are separate processes loading the data from S3 storage and preparing them for the master thread.
After our reasearch, we have come with following alternatives:
Feather
Parquet
ProtoBuf
JLD
JLD2
HDF5
Let’s discuss all options.
Feather is interesting, but it cannot handle unicode characters and sometime mysteriously crashes when reading large datafiles (we have filed bug for unicode characters).
We did not dare to try parquet, since the installation of the package is anything but trivial, and reading that using Spark.py means having separate Java process, which is something I want to avoid.
ProtoBuf is quite slow to read, even though we have fixed some type instabilities.
JLD contains memory leakage. The bug has been filed but due to transition to JLD2 is not addressed.
JLD2 crashes when saving large array of string (bug has been filed).
HDF5 seems to be the only format that is working (evaluation still under test, fingers crossed otherwise I am doomed and have to go to python, which is something I would rather avoid).
I have to say that this is not very nice situation. I like the Julia and being in a Python / Java environment is not a good feature. I think that having good and stable binary fileformat is important for any large-scale processing. Again, I am talking about terabyte scale data.
I do not know, where to move next. I am happy to help improving any package if my skills are sufficient, or at least do the testing.
Tomas
---
### Author:  [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)
#### Post date: [August 11, 2017, 11:15am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/2 "2017-08-11T11:15:23Z")
JLD2 would be the best first place for you to assist. I am not involved, but send a note via a JLD2 issue.
---
### Author:  [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)
#### Post date: [August 11, 2017, 1:25pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/3 "2017-08-11T13:25:03Z")
Check out [Parquet.jl](https://github.com/tanmaykm/Parquet.jl) – should be a bit easier to get going with than the java stack.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 11, 2017, 4:29pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/4 "2017-08-11T16:29:48Z")
> [@MikeInnes](#):
>
> Check out Parquet.jl – should be a bit easier to get going with than the java stack.
Parquet.jl is actually one of the examples that the original post was referring to as the poor state of this stuff, it currently is broken on julia 0.6 (see [Is this package still being maintained? · Issue #7 · JuliaIO/Parquet.jl · GitHub](https://github.com/JuliaComputing/Parquet.jl/issues/7)).
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [August 11, 2017, 7:52pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/5 "2017-08-11T19:52:35Z")
But updating Parquet.jl to work with Julia 0.6 may still be the easiest option.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [August 11, 2017, 8:10pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/6 "2017-08-11T20:10:14Z")
The installation of Parquet and its dependencies does not seem to be easy, and I have to say I am afraid of it. Meanwhile, we have tried flat-buffers and they seems to be OK. Reading is reasonably fast and it is possible to write the files straight from the java / scala. I just hope the implementation will be stable without memory leaks. Otherwise, the back-up solution is HDF5.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 11, 2017, 9:23pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/7 "2017-08-11T21:23:47Z")
> [@dfdx](#):
>
> But updating Parquet.jl to work with Julia 0.6 may still be the easiest option.
Yeah, I hope someone tackles that, if we had a working version of that, it seems it would be the best option.
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [August 11, 2017, 9:25pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/8 "2017-08-11T21:25:06Z")
> [@Tomas\_Pevny](#):
>
> The installation of Parquet and its dependencies does not seem to be easy
What operating system are you using? I’ve just got it working on Ubuntu 16.04 - with lots of warnings, but able to load a parquet file generated with Spark.
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [August 12, 2017, 12:01am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/9 "2017-08-12T00:01:03Z")
Made a couple of PRs to fix it:
- Zlib.jl: [https://github.com/dcjones/Zlib.jl/pull/32](https://github.com/dcjones/Zlib.jl/pull/32) (tests already passed with current master, I additionally fixed some depwarns)
- Thrift.jl: [https://github.com/tanmaykm/Thrift.jl/pull/28](https://github.com/tanmaykm/Thrift.jl/pull/28) (this should fix build on Ubuntu, macos should have been already supported, Windows is unlikely to come anytime soon)
- Parquet.jl: [https://github.com/JuliaComputing/Parquet.jl/pull/8](https://github.com/JuliaComputing/Parquet.jl/pull/8)
There are several more warnings when calling `using Parquet`, but all tests pass on my machine.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [August 12, 2017, 5:44am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/10 "2017-08-12T05:44:40Z")
Thanks a lot.
I wanted to point to this because it is something what my people drive-off julia.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 12, 2017, 9:02pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/11 "2017-08-12T21:02:32Z")
I looked at the Windows situation today. The first blocker seems to be that [Snappy.jl](https://github.com/bicycle1885/Snappy.jl/issues/12) doesn’t have Windows support, right?
With Thrift.jl, am I right that things might actually work even if the thrift compiler stuff doesn’t work on Windows? Isn’t that more of a dev time step to generate julia files, but that wouldn’t have to work on user machines, right? But, I didn’t look at it very long, so I might have completely misunderstood 🙂
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 12, 2017, 9:32pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/12 "2017-08-12T21:32:38Z")
I guess another alternative might be the other [Snappy.jl](https://github.com/krm01/Snappy.jl) package on github, which seems a pure julia implementation without any binary dependency, but it is not clear to me how [ready that one is](https://github.com/krm01/Snappy.jl/issues/1).
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [August 12, 2017, 11:44pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/13 "2017-08-12T23:44:57Z")
> [@davidanthoff](#):
>
> The first blocker seems to be that Snappy.jl doesn’t have Windows support, right?
Can you use Blosc instead?
---
### Author:  [@tkoolen](https://discourse.julialang.org/u/tkoolen)
#### Post date: [August 13, 2017, 2:22am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/14 "2017-08-13T02:22:54Z")
The JLD2 issue [appears to be fixed now](https://github.com/simonster/JLD2.jl/issues/31).
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 13, 2017, 4:51am UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/15 "2017-08-13T04:51:19Z")
> [@stevengj](#):
>
> Can you use Blosc instead?
Hm, that would be nice. The README of Blosc.jl says that the only algorithms currently supported are blosclz, lz4, and lz4hc, and I think for Parquet.jl we would need the Snappy algorithm. But the original (non-julia) library seems to support snappy, so maybe that is something that could be made to work.
>
>
> Snappy.jl currently doesn't work on Windows. Maybe https://github.com/stevengj/B…losc.jl could be used instead, at least the original underlying C version seems to support the snappy compression algorithm.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [August 13, 2017, 1:14pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/16 "2017-08-13T13:14:35Z")
> [@davidanthoff](#):
>
> I think for Parquet.jl we would need the Snappy algorithm.
I was wondering more whether it could use the Blosc algorithm itself.
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [August 13, 2017, 6:55pm UTC](https://discourse.julialang.org/t/the-poor-state-of-fileformats-for-high-performance-computing/5326/17 "2017-08-13T18:55:29Z")
> [@stevengj](#):
>
> I was wondering more whether it could use the Blosc algorithm itself.
You can add support for Parquet + Blosc, but to read existing files you still need to support Parquet + Snappy.