# Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project

**URL:** <https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209>\
**Category:** General Usage\
**Created:** [April 18, 2024, 3:58pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209 "2024-04-18T15:58:13Z")\
**Posts on this page:** 1\
**Showing post:** 34

<div class="post-metadata">

**Author:** ![Marcelo\_Simas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcelo_simas/32/208618_2.png) [@Marcelo\_Simas](https://discourse.julialang.org/u/Marcelo_Simas)\
**Post date:** [April 18, 2024, 11:38pm UTC](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209/34 "2024-04-18T23:38:01Z")

</div>

Just wanted to say that I’ve been able to stream a large arrow file (54 Gb, 4.5 million rows, 2,200 columns) using `Arrow.jl` + `TableOperations.jl` to generate complex statistical aggregations with `DataFrames.jl` while keeping memory usage around 1 Gb. The resulting solution ended up working faster than some C++ code which was using a proprietary binary format, but now a much wider audience can make improvements to that process.

Got a lot of valuable information on how to do that on this thread: [How well Apache Arrow’s zero copy methodology is supported? - Specific Domains / Data - Julia Programming Language (julialang.org)](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/25).

---

_[View the full topic](https://discourse.julialang.org/t/challenges-with-arrow-and-parquet-in-a-reasonably-substantial-julia-project/113209)._