# Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project

**URL:** <https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209>\
**Category:** General Usage\
**Created:** [April 18, 2024, 3:58pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209 "2024-04-18T15:58:13Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 8:48pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/21 "2024-04-18T20:48:37Z")

</div>

I am not asking for a solution, that’s why I am, well, not interested in a solution. Technically maybe this subtopic is better in the Community category, but anyways.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [April 18, 2024, 9:13pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/22 "2024-04-18T21:13:13Z")

</div>

JuliaIO needs a lot of work. In an ideal world, the packages would only rely on stable APIs and not need much maintenance. However, in my experience, many of the packages in JuliaIO rely on Julia internals to get around garbage collection issues or other performance issues. Many of the packages also suffer from the two-language problem because they wrap a library in another language, and those libraries are also often undermaintained, have bugs of their own, and tend to be overly complicated.

Right now I do all my IO with SmallZarrGroups.jl, ZipArchives.jl, and JSON3.jl and try and help keep those packages and their dependencies working.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [April 18, 2024, 9:18pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/23 "2024-04-18T21:18:01Z")

</div>

> Your problem. No point in discussing this in this forum.

well, it’s my problem too. I did the _exact same thing as OP_ and just used Polars to circumvent IO issues

not just with parquet or arrow. I have also had problems with `CSV.jl` to read (pretty innocent) csv files, and with the aws integrations for IO with s3

in general the “data engineering” corner of the world is prohibitively immature in Julia compared to python. again not blaming maintainers and I appreciate the time they spend on these packages. but it just is what it is

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 9:21pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/24 "2024-04-18T21:21:03Z")

</div>

For “curated” packages like those under JuliaIO, there should be no hacks. Just rock solid with mediocre performance. If I want something fast, I can always use some shiny third party packages.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [April 18, 2024, 9:31pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/25 "2024-04-18T21:31:00Z")

</div>

I think there is a bit of a miscommunication about what JuliaIO and similar organizations provide. They are just a convenience mechanism to share work between several people and de-risk in case a maintainer has to leave. These organizations don’t offer any particular guarantee of curation or quality.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [April 18, 2024, 9:32pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/26 "2024-04-18T21:32:52Z")

</div>

I don’t know anything about the Parquet format, but Parquet.jl depends on ThriftJuliaCompiler\_jll.jl which is a 3 year old fork of a 30K line C++ project [GitHub - tanmaykm/thrift: Mirror of Apache Thrift](https://github.com/tanmaykm/thrift?tab=readme-ov-file)

Is this heavy dependency needed? If I want to improve Parquet.jl how can I know if any of my changes will have some bad interaction with one of the 30K lines of C++.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 9:41pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/27 "2024-04-18T21:41:50Z")

</div>

Butbutbut… it has Julia in its title!!!

More seriously speaking, it’s kind of like an expectation problem. If Haskell can’t read Parquet or Arrow or have zombie libraries on them, I wouldn’t have any problem with it. I don’t expect Haskell to do that.

Julia is a number crunching language. It is designed from the ground up to do numerical computation. If I can’t input some data within 5 minutes of picking up the language, it’s a serious problem.

And do I want to contribute to the ecosystem? Yes, I contribute to packages, and sometimes even provide patches. This does not happen until I have a good experience first, because I am also doing it on my own free time.

There’s no free time when I am catching a deadline.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [April 18, 2024, 9:42pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/28 "2024-04-18T21:42:05Z")

</div>

First, I highly appreciate the free love authors put into Arrow.jl and Parquet.jl and Parquet2.jl.

I want to echo the observations that, things are lag behind and not nearly as feature complete in Julia world regarding Apache I/O. For the record, I only know [a few things](https://github.com/apache/arrow-julia/pull/419) about Arrow.jl from working on itself and [a similar format](https://indico.cern.ch/event/1222943/contributions/5144789/attachments/2580539/4450747/IRISHEP_RNTuple.pdf#page=28) #hep uses.

Over the years, I have filed a [few](https://github.com/apache/arrow-julia/issues/373) -[correctness](https://github.com/apache/arrow-julia/issues/427) - [problems](https://github.com/apache/arrow-julia/issues/484) or show-stopping bugs related to [performance](https://github.com/apache/arrow-julia/issues/244) and [large, compressed files](https://github.com/apache/arrow-julia/issues/413). I really, really want our Arrow story to be good, hell, I have [reverse](https://github.com/JuliaHEP/UnROOT.jl/pulls?q=is%3Apr+is%3Aclosed+%5BRNTuple%5D) engineered [something](https://github.com/root-project/root/blob/master/tree/ntuple/v7/doc/specifications.md) in #hep that’s as, if not more, complicated as Arrow.jl. So if there’s interests and positive feedback from maintainers, I’d love to throw in my free time in making Arrow.jl better.

I was very early to explore the zero-copy sharing with other Arrow implementation: [Re-use PyArrow memory via PyCall · Issue #92 · apache/arrow-julia · GitHub](https://github.com/apache/arrow-julia/issues/92#issuecomment-1232391859). And we still want to see that happen especially now with efforts like AwkwardArray.jl

Anyway, all of these went nowhere, so, at the moment, it’s hard for me to say, but I feel powerless to help the situation.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [April 18, 2024, 9:47pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/29 "2024-04-18T21:47:06Z")

</div>

For “big data” IO formats such as parquet, and to some extent arrow, I think the long term solution is to have a solid, reliable, and straightforward way to wrap rust libraries in Julia. While it’s great to have whatever effort the community puts into them, the reality is that most Julia developers (including myself) are here for other reasons and not particularly excited about maintaining enterprise IO formats. I am happy to continue maintaining Parquet2.jl, and will fix what issues I can, but it is also a huge format with a bewildering set of features and much of what writes parquets are JVM packages that probably aren’t particularly interested in interop with stuff that’s not JVM. The arrow standard provides the ability to provide low-level buffer views of data in wrapped libraries, so there is every reason to wrap `polars` and take advantage of all the work that is happening there to deal with this stuff. Many Julia developers have at least some interest in rust, and some great work has been done with [`jlrs`](https://github.com/Taaitaaiger/jlrs) but it would be nice to have more people to continue the effort. There has also been [`Polars.jl`](https://github.com/Pangoraw/Polars.jl) which seems functional, but as far as I know does not do low-level wrapping of arrow buffers, so its applicability may be limited.

At the same time, I would encourage new users to be open to calling dependencies from other languages if needed. This is how relatively new and niche languages are able to establish themselves in the first place. While there may be few to no benefits to using Julia if all you’re doing is taking the output of one black box wrapped function and plugging it into another, and there are sometimes real obstacles to using wrapped packages (such as the difficulties with using `pyarrow` that motivated me to write Parquet2.jl), there are also a huge number of cases where you can use a dependency for one specific thing that you may not have native support for and it isn’t a big deal.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 9:50pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/30 "2024-04-18T21:50:54Z")

</div>

Right, I mentioned in the original topic, but I think I should restate here:

Julia’s community is awesome. I appreciate all the work put into all these community packages so I can simply type `]add` to access all of them.

I do not mean to talk bad things about these packages, as they no doubt required lots of effort from authors and contributors. However, practically speaking, I cannot base my projects on an unstable foundation.

I’ve had the pleasure to debug a full-stack problem where a CSS style sheet entry went through a whole stack of software and caused my desktop environment to crash when I type in one specific window using a specific Chinese input method. I do not want to repeat this on my primary research project.

I think it’s better to have a solid, low-level API than attempting to build something high-level and abstracted. There’s far less work, it allows users to circumvent more bugs by hacking around them, and it serves as a good intermediate API.

Something like a thin wrapper above `_jll` libraries, just barely making the API julia-like would be a good start.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [April 18, 2024, 9:54pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/31 "2024-04-18T21:54:37Z")

</div>

> [@rongcuid](#):
>
> Something like a thin wrapper above `_jll` libraries, just barely making the API julia-like would be a good start.

The practical problem with that is Arrow reference implementation is in C++ not in C, so that’s immediately non-trivial. We have \_jll wrappers like that for other applications, especially when the target/reference is in C. Generally I don’t think people are avoiding \_jll wrappers for “I want pure Julia”, but when it’s C++ it’s a different ball game

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 9:59pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/32 "2024-04-18T21:59:03Z")

</div>

Ok, I know C++ FFI is difficult.

On a side note, I think Arrow’s documentation is pretty bad as well.

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [April 18, 2024, 10:24pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/33 "2024-04-18T22:24:46Z")

</div>

> [@ExpandingMan](#):
>
> For “big data” IO formats such as parquet, and to some extent arrow, I think the long term solution is to have a solid, reliable, and straightforward way to wrap rust libraries in Julia. While it’s great to have whatever effort the community puts into them, the reality is that most Julia developers (including myself) are here for other reasons and not particularly excited about maintaining enterprise IO formats.

I think that would likely come from a group that was entrepreneurial enough to create a commercial ETL framework to use Julia for specialized or udf code, and reuse the rust/c++ libraries for operations which are common across languages.

---

<div class="post-metadata">

**Author:** ![Marcelo\_Simas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcelo_simas/32/208618_2.png) [@Marcelo\_Simas](https://discourse.julialang.org/u/Marcelo_Simas)\
**Post date:** [April 18, 2024, 11:38pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/34 "2024-04-18T23:38:01Z")

</div>

Just wanted to say that I’ve been able to stream a large arrow file (54 Gb, 4.5 million rows, 2,200 columns) using `Arrow.jl` + `TableOperations.jl` to generate complex statistical aggregations with `DataFrames.jl` while keeping memory usage around 1 Gb. The resulting solution ended up working faster than some C++ code which was using a proprietary binary format, but now a much wider audience can make improvements to that process.

Got a lot of valuable information on how to do that on this thread: [How well Apache Arrow’s zero copy methodology is supported? - Specific Domains / Data - Julia Programming Language (julialang.org)](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/25).

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 11:42pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/35 "2024-04-18T23:42:03Z")

</div>

I tried `TableOperations` when I was doing the project in Julia. Its documentation on streaming tables was incomprehensible. At the end, Polars’s query language was just much more powerful.

That Polars-python itself had a chance of producing wrong results in Jupyter was story of another time…

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [April 18, 2024, 11:55pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/36 "2024-04-18T23:55:48Z")

</div>

I like `OnlineStats.jl` quite a lot. a very smooth integration with streaming arrow to that package would be fantastic

---

<div class="post-metadata">

**Author:** ![asbisen](https://avatars.discourse-cdn.com/v4/letter/a/a9a28c/32.png) [@asbisen](https://discourse.julialang.org/u/asbisen)\
**Post date:** [April 19, 2024, 12:32am UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/37 "2024-04-19T00:32:49Z")

</div>

A sidebar question. Is C++ FFI difficult only from Julia or also Python?

---

<div class="post-metadata">

**Author:** ![Marcelo\_Simas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcelo_simas/32/208618_2.png) [@Marcelo\_Simas](https://discourse.julialang.org/u/Marcelo_Simas)\
**Post date:** [April 19, 2024, 12:38am UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/38 "2024-04-19T00:38:41Z")

</div>

I got better guidance from forum posts than the actual docs. I feel like they could have more examples and guidance. I am hopeful things are moving in the right direction and will start getting more engaged and try to help as much as possible. I’ve been an R user for close to 15 years and have had a complex love hate relationship with Python over much of that time and doing algorithm implementation work in Julia felt much more natural and empowering.

I do agree that investment needs to be done on supporting read/write to Arrow and Parquet as those are key to any serious data processing solution going forward.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 19, 2024, 12:58am UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/39 "2024-04-19T00:58:50Z")

</div>

C++ FFI is difficult everywhere, because their names are mangled.

Lingua franca of FFI is C, because its ABI is pretty much the _de facto_ standard of every cross-language calling convention.

All machines speak C, basicaly.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 19, 2024, 1:03am UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/40 "2024-04-19T01:03:24Z")

</div>

I hate Python, but its IO is so good.

It’s hard to maintain, its module system is lacking, I dislike its syntax, its package management is abyssmal…

However, it’s has a large stdlib, it runs external programs easily, it has **so many tutorials** in every single module. Any beginner with a hint of CS knowledge can learn it by just reading the official docs. It’s so easy to interface with other tools. It handles many file formats in idiomatic Python…

At some point there are just so many beginners attracted that quantity becomes a quality.

[Previous page](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209.md?page=1)

[Next page](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209.md?page=3)