# Writing Parquet files
**URL:** https://discourse.julialang.org/t/writing-parquet-files/11221
**Category:** General Usage
**Created:** [May 29, 2018, 7:49am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221 "2018-05-29T07:49:48Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [May 29, 2018, 7:49am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/1 "2018-05-29T07:49:48Z")
Is it possible to write Parquet files from Julia? I found [Parquet.jl](https://github.com/JuliaComputing/Parquet.jl) but looks like it can read but not write…
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [May 29, 2018, 10:46am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/2 "2018-05-29T10:46:27Z")
Not the most optimal way, but [Spark.jl](https://github.com/dfdx/Spark.jl) contains [methods](https://github.com/dfdx/Spark.jl/blob/master/src/sql.jl#L63-L72) for reading and writing Parquet files. The downside is dependency on Java, of course.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [May 29, 2018, 1:39pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/3 "2018-05-29T13:39:01Z")
Good call… but it requires a spark dataset to write Parquet file. I’m trying to find a more efficient way to push data to Spark, a chicken and egg problem…
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 29, 2018, 2:10pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/4 "2018-05-29T14:10:47Z")
Even in Python it seems that people always deal with parquets using spark. This does seem like ridiculous overkill depending on what you’re doing, but my impression is that parquets were designed for enormous datasets.
It still seems like a bad solution since there appears to be [no direct way](https://stackoverflow.com/questions/38271025/how-to-save-file-in-feather-format-storage-from-spark) of loading feather files to spark, but you might check out [feather](https://github.com/JuliaData/Feather.jl).
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [May 29, 2018, 2:28pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/5 "2018-05-29T14:28:54Z")
My data set isn’t very big - just 60G. It seems that the most reliable way to load data into spark is CSV. however, CSV is lossy for Float64 so I’m looking for a binary format. Compression is best since I need to upload the file to spark via the net since the environment is i the cloud.
I’m aware of Feather.jl and I love it!
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 29, 2018, 2:43pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/6 "2018-05-29T14:43:39Z")
Yeah, the binary data format situation is bizarrely terrible.
From where I sit right now in the private sector I can tell you that a big part of the reason this is the case is because of a staggering amount of horribly formatted legacy data, exacerbated by people who don’t know any better constantly outputting everything in sight to (often corrupted) csv’s, or, even worse excel sheets 😱 (yes, there are people out there who even seem to think it’s a good idea to have a 500MB excel sheet). Explaining the concept of double precision floats to those people is downright impossible.
Anyway, the feather standard will ultimately work for large datasets by linking together 4GB files. Unfortunately it seems that the feather authors have been dragging their feet and don’t have any kind of standard yet (that I know of). If this doesn’t happen before long, we will come up with our own temporary solution for Feather.jl.
You might look to see if there is some sort of protobuf related solution, I know that in many cases data is stored en masse in protobufs, but in the examples I know of this was somehow done _inside_ of a parquet.
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [May 29, 2018, 3:46pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/7 "2018-05-29T15:46:21Z")
I’ve been thinking more about this recently; I think we might be better off just committing more to Arrow.jl and it’s format; there really isn’t any long-term need, IMO, for feather since Arrow is the actual format layout definition, _including_ binary tranfser. There’s no reason you couldn’t just write the Arrow format to disk and use that instead of feather.
The one advantage that Parquet includes over something like Arrow, at least as far as I understand the implementations currently, is that Parquet includes native support for compression.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 29, 2018, 4:53pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/8 "2018-05-29T16:53:46Z")
> [@quinnj](#):
>
> I think we might be better off just committing more to Arrow.jl and it’s format; there really isn’t any long-term need,
The thing we are really missing right now is an expanded metadata for Feather. Right now it only supports single files up to 4GB, we need the extension to the metadata that will support groups of Feather files.
The important thing missing from Arrow.jl right now is a “concatenation wrapper”: i.e. an `AbstractArray` that joins together multiple underlying arrow arrays. As far as I can tell this is the only additional arrow component needed to support large feather files.
Parquet already [complies](https://arrow.apache.org/docs/python/parquet.html) with the arrow standard somehow. I don’t know how it handles compression, but I assume that there is some standard for compressing arrow data that is not well documented. I’ve thought about writing a package that uses the Arrow.jl back-end that reads and writes parquets, but given that [Parquet.jl](https://github.com/JuliaComputing/Parquet.jl) exists, is well maintained and already uses memory mapping for lazy loading, my motivation to do that is approximately 0 (it would be a lot of work).
@quinnj, I think that the metadata aspect of Arrow is sufficiently vague that there actually is a very significant difference between different arrow compliant formats. After all, feather and parquet are quite different. As you can see the current Feather.jl code is not quite trivial, though I like to think I’ve gone to great lengths to simplify it as much as possible and to make future metadata layers on top of Arrow.jl as simple as possible (e.g. the “`Locate`” interface).
---
### Author:  [@piever](https://discourse.julialang.org/u/piever)
#### Post date: [May 29, 2018, 6:15pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/9 "2018-05-29T18:15:53Z")
One limitation I found working with Feather is that it could not save/load non-bits types (meaning, if a column of the dataset had non bits types, it would error). Would Parquet also have the same limitation? If not, that may be a reason for investing effort in the Parquet format to read/write data.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 29, 2018, 6:22pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/10 "2018-05-29T18:22:44Z")
> [@piever](#):
>
> Would Parquet also have the same limitation?
Yes. These formats are not intended for serializing arbitrary Julia data. In principle once we implement arrow structs we will give you some kind of way of doing this, but it still probably wouldn’t be your best option for that. I think what you are looking for is [JLD2.jl](https://github.com/simonster/JLD2.jl).
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [May 29, 2018, 9:37pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/11 "2018-05-29T21:37:25Z")
I wonder if I could just use PyCall to hook up with [fastparquet](https://pypi.org/project/fastparquet/)? Itching to try…
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 29, 2018, 10:41pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/12 "2018-05-29T22:41:30Z")
Looks like you’ll basically have 3-way serialization overhead (Julia → Pandas → Parquet) but might as well give it a try.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [May 31, 2018, 3:59am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/13 "2018-05-31T03:59:20Z")
So it does work. In my test, for a 17 GiB data set, it only takes few seconds for Julia → Pandas but Pandas → Parquet with GZIP took over 1 hr 15 mins. I guess Python isn’t very fast there…
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [May 31, 2018, 11:09am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/14 "2018-05-31T11:09:39Z")
Amen to the discussion on CSV files. At my last employ I discovered that one set of engineers were working with a directory which had thousands of CSV files. These turned out to be images of semiconductor wafers which were being processed, along a time axis.
In my experience engineeers just want to get on and do their work, and their managers are generally breathing down their necks to get results. So if one person implements a workflow which uses CSV files, the next person then the next will come along and repeat that.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 31, 2018, 1:57pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/15 "2018-05-31T13:57:52Z")
> [@tk3369](#):
>
> it only takes few seconds for Julia -\> Pandas but Pandas -\> Parquet with GZIP took over 1 hr 15 mins. I guess Python isn’t very fast there…
To be fair the Julia → Pandas probably isn’t very much actual work, depending on your data. In fact, if all you had were `Vector`s of integers and floats, the Julia and numpy memory layouts would be identical, so it would be instantaneous in that case. The GZIP compression on the other hand may actually have to do quite a lot of work, depending on which compression algorithm it’s using, though I don’t understand why it took quite that long.
> [@johnh](#):
>
> These turned out to be images of semiconductor wafers which were being processed, along a time axis.
That’s a great (but scary) story! 😆
A major part of the problem with the csv’s is that there aren’t really that many sensible alternatives. HDF5 is fine, but there never seems to be any consensus on how to lay it out. As far as I know there aren’t even any truly “standard” HDF5 tabular formats. Parquet is increasingly popular, but it does seem very much geared toward huge datasets, and I know that with it’s many separate files it can sometimes be a burden on the file system.
When I was in high energy physics we used [ROOT trees](https://root.cern/trees-five-steps) for storing data. It was a standard practice that nobody ever questioned and it worked just fine. When I came to doing data science in the private sector, I was _shocked_ that there seemed to be no standard non-HEP alternative to this. When I realized it was common practice to store a dataset in a 10GB csv I thought “what have I gotten myself into?”.
This is why I’ve done so much work on [Feather.jl](https://github.com/JuliaData/Feather.jl) which has lots of really nice features now and we will announce some time after the official 0.7 release, though the maintainers of the Feather standard have yet to decide on metadata formats for large datasets (we will come up with our own if they don’t).
---
### Author:  [@ValdarT](https://discourse.julialang.org/u/ValdarT)
#### Post date: [January 26, 2019, 7:19am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/16 "2019-01-26T07:19:32Z")
> [@ExpandingMan](#):
>
> I’ve thought about writing a package that uses the Arrow.jl back-end that reads and writes parquets, but given that [Parquet.jl](https://github.com/JuliaComputing/Parquet.jl) exists, is well maintained and already uses memory mapping for lazy loading, my motivation to do that is approximately 0
There’s still no support for _writing_ parquets in Parquet.jl though. Nor anywhere else (except through Spark) currently afaik.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [May 9, 2020, 12:09pm UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/17 "2020-05-09T12:09:12Z")
I got a functional parquet writer here
> **[GitHub - xiaodaigh/Diban.jl: Dìbǎn (地板) is a Parquet reader and writer](https://github.com/xiaodaigh/Diban.jl)**
>
> Dìbǎn (地板) is a Parquet reader and writer. Contribute to xiaodaigh/Diban.jl development by creating an account on GitHub.
Please help test out Diban.jl, I will start work on a PR to Parquet.jl, I really don’t want the community to have two separately maintained parquet package, but I also don’t see why ppl should wait for something that can be done right away. So I will take a two pronged approach where I will try to PR to Parquet.jl and if it takes too long, I will publish Diban
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [May 28, 2020, 10:34am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/18 "2020-05-28T10:34:51Z")
Sorry for necro-posting but there is a parquet writer in Parquet.jl now [Release v0.6.0 · JuliaIO/Parquet.jl · GitHub](https://github.com/JuliaIO/Parquet.jl/releases/tag/v0.6.0)
It’s based on the Diban work
---
### Author:  [@merlin](https://discourse.julialang.org/u/merlin)
#### Post date: [October 2, 2020, 2:31am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/19 "2020-10-02T02:31:03Z")
I just want to thank you for this work, which apparently instigated the addition feature.
Its becoming somewhat of an industry standard to store datasets on s3 and run data pipelines via Spark jobs, before writing again to s3 (or the google/azure equiv).
The format is commonly parquet, which is convenient because:
- columnn store gives faster read access on transformations.
- partitioning is inherent to the file structure, again for more direct access to relevant data
- the schema is stored with the objects, including types, so there is no laborious schema definition phase to reading it
- files are written in chunks for better parallel loading
- array type columns, which is commonly used to store image or text embedding data along with other records
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [October 2, 2020, 4:45am UTC](https://discourse.julialang.org/t/writing-parquet-files/11221/20 "2020-10-02T04:45:45Z")
thanks @merlin I got some funding from NumFocus too.
[Next page](https://discourse.julialang.org/t/writing-parquet-files/11221.md?page=2)