# Arrow, Feather, and Parquet

**URL:** <https://discourse.julialang.org/t/arrow-feather-and-parquet/28739>\
**Category:** Data\
**Tags:** parquet, arrow\
**Created:** [September 13, 2019, 8:43pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739 "2019-09-13T20:43:18Z")\
**Posts on this page:** 9\
**Page:** 3

<div class="post-metadata">

**Author:** ![wsphillips](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsphillips/32/6777_2.png) [@wsphillips](https://discourse.julialang.org/u/wsphillips)\
**Post date:** [December 2, 2019, 10:01pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/41 "2019-12-02T22:01:26Z")

</div>

I had seen those, but both looked crufty/neglected…really hoping something fresh and more friendly to non-MPI methods of distributed computing pans out. HDF5 has always been there, but the codebase is literally millions of lines and its maintained by a small number of developers (bless their souls), so updates to the standard come very slowly and very incrementally. Hard for it to keep up with changing landscape of data science needs (unfortunately!)

---

<div class="post-metadata">

**Author:** ![ihnorton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihnorton/32/26_2.png) [@ihnorton](https://discourse.julialang.org/u/ihnorton)\
**Post date:** [December 9, 2019, 7:48pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/42 "2019-12-09T19:48:52Z")

</div>

> [@wsphillips](#):
>
> I think TileDB storage with (optionally) Arrow in-memory objects could do some nifty things

@wsphillips thanks for the mention of TileDB. This thread and some of the links were very interesting to read. @wesm makes the important point that existing columnar and scientific data formats were designed to solve distinct problems:

> [@wesm](#):
>
> comparing HDF5 with a columnar file format is truly apples and oranges.

However, with TileDB we aim to bridge the gap between scientific and columnar formats with a single efficient, cloud-optimized array format. TileDB provides fast dense array support, solving various limitations of HDF5 (e.g., parallel writes). But TileDB can also model dataframes with its sparse array support, by selecting a subset of its columns to represent the dimensions (see [this discussion](https://docs.tiledb.com/main/use-cases/dataframes)). Our integrations with Spark, MariaDB and PrestoDB show how we can feed TileDB arrays into SQL-based query engines. We are currently working to round out TileDB’s dataframe capability with heterogenous dimension types, support for string dimensions, and predicate push-down, which will be released in the next few months.

We’ve recently [discussed](https://medium.com/tiledb/tiledb-a-database-for-data-scientists-ddf4ca122176) our vision to simplify data science through the integration of careful format design and efficient storage engine library, capable of updates, partitioning logic, and optimizations. For those not familiar with TileDB: in a nutshell, it consists of a fast, open-source, C++ library that is fully parallelized for dense and sparse array i/o without relying on separate libraries (Dask, Spark, etc.), and works particularly well on AWS S3. We aim to make this technology available as broadly as possible by continuing to build efficient high-level APIs (currently: C, C++, Python, R, Java, Go) and integrations (currently: Dask, Spark, PrestoDB, MariaDB, PDAL, GDAL). Regarding Arrow specifically, we recently used it in our VCF genomics library (see [here](https://github.com/TileDB-Inc/TileDB-VCF/blob/master/apis/python/src/tiledbvcf/binding/reader.cc#L298)), and based on that positive experience, we are planning on adding Arrow support to the core library as well.

---

<div class="post-metadata">

**Author:** ![wsphillips](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsphillips/32/6777_2.png) [@wsphillips](https://discourse.julialang.org/u/wsphillips)\
**Post date:** [December 9, 2019, 8:00pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/43 "2019-12-09T20:00:39Z")

</div>

> [@ihnorton](#):
>
> Regarding Arrow specifically, we recently used it in our VCF genomics library (see [here](https://github.com/TileDB-Inc/TileDB-VCF/blob/master/apis/python/src/tiledbvcf/binding/reader.cc#L298)), and based on that positive experience, we are planning on adding Arrow support to the core library as well.

Very cool. Also for those interested, I also had some extremely thorough and helpful feedback from the Zarr team about where they are going with the new version 3.0 spec and how Zarr relates to TileDB and other packages in this niche: [Question/possible enhancement: Relationship to N5 + Arrow/Parquet? · Issue #515 · zarr-developers/zarr-python · GitHub](https://github.com/zarr-developers/zarr-python/issues/515)

@ihnorton Anything you want to add/clarify about the comments there?

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 9, 2019, 9:43pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/44 "2019-12-09T21:43:33Z")

</div>

Anyone have experience with streaming copy-free arrow in Julia?

I heard this is being done in python and in a few other languages, and was curious to try it myself.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 9, 2019, 10:16pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/45 "2019-12-09T22:16:52Z")

</div>

I only read recently that it isnt possible yrt

---

<div class="post-metadata">

**Author:** ![ihnorton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihnorton/32/26_2.png) [@ihnorton](https://discourse.julialang.org/u/ihnorton)\
**Post date:** [December 10, 2019, 5:49pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/46 "2019-12-10T17:49:54Z")

</div>

> [@wsphillips](#):
>
> Anything you want to add/clarify about the comments there?

That was a nice response, thanks for fostering this exchange.

The core architectural choices predate my involvement with TileDB by several years, but Stavros (TileDB’s [author](https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf)) just posted a [response outlining](https://github.com/zarr-developers/zarr-python/issues/515#issuecomment-564141789) how some of those technical decisions were driven by TileDB’s original motivating use-case:

> 1. allowing sample additions to sparse arrays over time – for genomic variant calling specifically, the “N+1” problem – offering rapid updates
> 2. allowing space, time, and i/o efficient queries on 10M+ ranges on a 100TB sparse array (this is a realistic dataset size for our largest genomics users)
> 3. allowing versioning (“time traveling”) of un-consolidated datasets, which is important for auditability, database interaction, etc.

---

<div class="post-metadata">

**Author:** ![rabernat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rabernat/32/11890_2.png) [@rabernat](https://discourse.julialang.org/u/rabernat)\
**Post date:** [December 18, 2019, 8:18pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/47 "2019-12-18T20:18:05Z")

</div>

For anyone who wants to try out Zarr.jl with data stored in Google Cloud, here is a repo / binder with some examples:

[https://github.com/pangeo-data/pangeo-julia-examples/](https://github.com/pangeo-data/pangeo-julia-examples/)

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [September 20, 2020, 10:44pm UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/48 "2020-09-20T22:44:34Z")

</div>

Would the new Arrow C [data](https://github.com/apache/arrow/blob/master/docs/source/format/CDataInterface.rst) and [stream](https://github.com/apache/arrow/pull/8052) interfaces help with ease of implementation?

---

<div class="post-metadata">

**Author:** ![mks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mks/32/18374_2.png) [@mks](https://discourse.julialang.org/u/mks)\
**Post date:** [November 1, 2020, 11:02am UTC](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739/49 "2020-11-01T11:02:16Z")

</div>

An interesting in-memory shared object storage provided by Apache Arrow is the [Plasma storage](https://ray-project.github.io/2017/08/08/plasma-in-memory-object-store.html)

I couldn’t yet find a Julia implementation, so I thought I’d mention it here to raise some evaluation interest.

As an example in Python, there is this project, that adds a namespace to the Plasma storage here: [https://github.com/russellromney/brain-plasma](https://github.com/russellromney/brain-plasma)

[Previous page](https://discourse.julialang.org/t/arrow-feather-and-parquet/28739.md?page=2)