# Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project

**URL:** <https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209>\
**Category:** General Usage\
**Created:** [April 18, 2024, 3:58pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209 "2024-04-18T15:58:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris-b1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris-b1/32/14165_2.png) [@chris-b1](https://discourse.julialang.org/u/chris-b1)\
**Post date:** [April 18, 2024, 3:58pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/1 "2024-04-18T15:58:13Z")

</div>

> [@Experience report after finishing a (reasonably substantial) Julia project in 2024](https://discourse.julialang.org/t/experience-report-after-finishing-a-reasonably-substantial-julia-project-in-2024/113148/1):
>
> For other formats, Julia significantly lags behind other languages like R or Python. Recently popular are Parquet and Arrow IPC, which provide efficient binary formats for exchanging large amount of data. However, Julia’s support for either is _terrible_. Despite being under `JuliaIO` organization, these important (I think) libraries remain unmaintained and unfinished. This really tanks my confidence on the organization, honestly.
> 
> I know better to not use Arrow or Parquet in this project, because I know how bad their support are in Julia, but the primary reason I got rid of Julia from all my research code, was that it cannot work with these files properly.

I was curious if you could expand on this a bit. In my experience Arrow.jl is well supported and performant.

I haven’t used it much but there is also Parquet2.jl which is actively developed.

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [April 18, 2024, 4:21pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/2 "2024-04-18T16:21:14Z")

</div>

Arrow.jl does not support the C interface so it’s not possible to zero copy an arrow object between languages. It also doesn’t support ADBC or Arrow Flight, which I think would be really beneficial to allow users to host a Flight server with Julia as a backend.

For all IO, I have found using DuckDB.jl and its support across other languages to be helpful, although its client api is not officially supported.

Instead of JSON + StructTypes, I have found Serde.jl to be helpful.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 4:21pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/3 "2024-04-18T16:21:16Z")

</div>

`Arrow.jl` doesn’t support streaming correctly. It also reads the entire arrow data into memory, despite the file being uncompressed. I had some other issues which I don’t remember that forced me to first switch to `Parquet` because at least Parquet was supported by Julia, then even `Parquet.jl` broke. Then I switched to `DuckDB.jl`, and alas, duckdb 0.9.1 segfaulted only on Julia. By that time I just switched to Python and Rust fully.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [April 18, 2024, 6:43pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/4 "2024-04-18T18:43:09Z")

</div>

> [@Experience report after finishing a (reasonably substantial) Julia project in 2024](https://discourse.julialang.org/t/experience-report-after-finishing-a-reasonably-substantial-julia-project-in-2024/113148/1):
>
> I know better to not use Arrow or Parquet in this project, because I know how bad their support are in Julia

Are you talking about Parquet2.jl and Arrow.jl? What problems did you have?

Edit: This is discussed [above](https://discourse.julialang.org/t/experience-report-after-finishing-a-reasonably-substantial-julia-project-in-2024/113148/22) already.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 6:46pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/5 "2024-04-18T18:46:26Z")

</div>

`Arrow.jl` and `Parquet.jl`.

`Arrow.jl` has:

- No memory-mapped reads
- No real streaming (doc says it has, but it actually reads the whole table in memory)

`Parquet.jl` has:

- Can’t read my data, period.

I didn’t know `Parquet2` before I switched language. I mean, `Parquet.jl` is supposed to be maintained by `JuliaIO`, so I never looked elsewhere.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [April 18, 2024, 6:47pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/6 "2024-04-18T18:47:40Z")

</div>

well, `Parquet2.jl` cannot read my parquet file 🙂

through no fault of @ExpandingMan — I appreciate his prompt attention when I have submitted issues — I guess it is due to some known issue in `CodecLz4.jl`

but the fact remains that I cannot really use Julia for this task

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 7:04pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/7 "2024-04-18T19:04:53Z")

</div>

Huh, is thread-splitting some discourse magic I am not aware of?

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [April 18, 2024, 7:07pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/8 "2024-04-18T19:07:13Z")

</div>

Well, I am a happy user of Arrow.jl… Much faster to store and read files in .arrow format than doing that with .csv files…

I admit, I don’t have files larger than my RAM…

But I think reading files larger than your RAM should work…

_One immediate, though perhaps not incredibly useful example, is Arrow.Stream.  
Arrow.Stream implements Tables.partitions in that it iterates “tables” (specifically Arrow.Table), and as such, Arrow.write will iterate an Arrow.Stream, and write out each Arrow.Table as a separate record batch._

If it doesn’t work for you, please provide an MWE that reproduces the problem.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 7:18pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/9 "2024-04-18T19:18:40Z")

</div>

I don’t remember exactly why I dropped `Arrow.jl`. Either streaming straight up didn’t work, or working with `Table.partitions` was too tricky.

I was dealing with some IPC files that were nearly 30GB, with zstd compression.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [April 18, 2024, 7:21pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/10 "2024-04-18T19:21:14Z")

</div>

Well, if you say something doesn’t work or is too tricky without asking for a solution here on Discourse, well, this is really your problem, and not a problem of Julia or the Julia ecosystem…

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 7:25pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/11 "2024-04-18T19:25:27Z")

</div>

Well, between “figuring out myself”, “asking questions”, “trying another format”, and “switch to Python and have it working right now”, I simply chose the last.

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [April 18, 2024, 7:43pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/12 "2024-04-18T19:43:04Z")

</div>

There has only been three commits in the six months to Arrow.jl since the repo moved under Apache.

Apache arrow asked for [examples](https://github.com/apache/arrow/issues/40465) from each language on how to serve arrow via http, and almost all languages except Julia have been completed.

I have never seen arrow passed to Rcall or python via zero copy in Julia.

There is more support in other languages than just reading and writing ipc files.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [April 18, 2024, 8:20pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/13 "2024-04-18T20:20:28Z")

</div>

> [@rongcuid](#):
>
> I simply chose the last.

This indicates that you do not have the need to write fast code yourself, that you are happy what Python libraries provide… No problem, but if I would write my code in Python it would be 10 to 100 times slower than the code I write in Julia, and THAT would be a problem for me…

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [April 18, 2024, 8:29pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/14 "2024-04-18T20:29:08Z")

</div>

OP said they’re also using Rust, which is famously fast.

I think our best option here is to thank OP for their detailed experience report and try to learn what we can from it.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 8:30pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/15 "2024-04-18T20:30:13Z")

</div>

Python is never used by itself. In this case, I used Polars, which is written in Rust and gave a python API. It is just as fast as if I used Julia.

In my original topic, I said that Julia is great if I need fast user defined function, and it shows when I needed to write UDF for Polars that goes through pycall, which was very slow. However, if I use Julia by itself, IO is a big problem.

Running some slow python code is still faster than debugging something as fundamental as an IO package. Here’s something I think is very important for me:

I DON’T want to deal with IO problems. If I can’t even import data correctly just by reading documentation, it’s out. I lose trust.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [April 18, 2024, 8:38pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/16 "2024-04-18T20:38:02Z")

</div>

> [@rongcuid](#):
>
> I DON’T want to deal with IO problems. If I can’t even import data correctly just by reading documentation, it’s out. I lose trust.

Your problem. No point in discussing this in this forum.

---

<div class="post-metadata">

**Author:** ![rongcuid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rongcuid/32/42911_2.png) [@rongcuid](https://discourse.julialang.org/u/rongcuid)\
**Post date:** [April 18, 2024, 8:38pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/17 "2024-04-18T20:38:27Z")

</div>

No one is forcing you to reply.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [April 18, 2024, 8:44pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/18 "2024-04-18T20:44:34Z")

</div>

> [@ufechner7](#):
>
> Your problem. No point in discussing this in this forum.

This forum is exactly the sort of place where we can discuss our Julia problems, and benefit from the help of others. Let’s keep it as welcoming and useful as we can.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [April 18, 2024, 8:46pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/19 "2024-04-18T20:46:32Z")

</div>

Well, there is no point in discussion problems with people who are not looking for a solution…

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [April 18, 2024, 8:47pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/20 "2024-04-18T20:47:40Z")

</div>

If you think this discussion is pointless you are free to ignore it. But please don’t discourage others from asking questions or providing answers

[Next page](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209.md?page=2)