Continuing the discussion from JDF - an experimental DataFrame serialization format is ready for beta testing : @ExpandingMan Thank you very much! That’s a great summary and it helps me getting idea of reliability level of feather format. It’s also nice to get a summary of the recent activities in A…

[image] xiaodai: The post technically is talking about Arrow (which I think is closely related to parquet). I found the situation quite confusing myself and believe I have pieced it together over time. My understanding is that the original goal of arrow was to provide a common in-memory data…

Would be great if you could open an issue, no chance of it getting fixed otherwise :slight_smile:

It’s rather confusing for me too. I know that Arrow is in-mem and Parquet is on disk. But what’s giving the impression that they getting tightly tangled is that pyarrow provides the read_parquet functionalties for pandas and R’s {arrow} has the only reliable read_parquet function in native R (i.e. n…

As for Arrow itself, the conference post you shared was interesting, and I certainly share some of their apparent consternation about it. As for C++ idioms, I did not spend too much time looking at their code, and though I have a good deal of experience in C++, I was working in the high energy phys…

[image] xiaodai: They sit together in both Python and R, so must be related somehow is my logic. Like I said, you can (and they do) create a parquet reader that emits arrow messages. That’s what’s going on there, take a look at the parquet file format . It’s different than arrow, it even use…

[image] ExpandingMan: The format is extremely general and supports quite a few composite data types. I therefore don’t really understand why it was decided to keep the format so focused on tabular data. It reminds me of this talk ["Particle physics, 10,000 times faster" by Jim Pivars…

[image] tkf: IIRC it was about particle physics people using Arrow as a building block for non-tabular data structures. Wow, very interesting, this talk bridges the phases of my adult life lol. I find it really disheartening how Python seems to be working its way more deeply into HEP world. …

My workshop from VLDB might be helpful in clearing up some of the confusion about how Arrow’s binary protocol works [image] GitHub - wesm/vldb-2019-apache-arrow-workshop: Materials for Apache Arrow... Materials for Apache Arrow workshop at VLDB 2019. Contribute to wesm/vldb-2019-apache-…

Hi, I would like to have an opportunity to address your complaints and clear up your confusion. Can you please write to dev@arrow.apache.org ? The slide deck I posted above should also help address confusion as it gives a deep dive into the columnar format. I have also recently cleaned up the specifi…

So I shouldn’t be using “.feather” anymore?

You can certainly keep using Feather files. The internal details of Feather files are going to change in the next 12 months to take advantage of the work we have done in the Arrow project, but we will maintain compatibility code for some period of time to read old files.

Arrow, Feather, and Parquet

Specific Domains Data

tkf September 16, 2019, 7:10pm 21

It reminds me of this talk

IIRC it was about particle physics people using Arrow as a building block for non-tabular data structures.

Topic		Replies	Views
Writing Parquet files General Usage	28	5512	November 12, 2020
What fileformat to use to load data for high performance computing Machine Learning	37	7261	December 1, 2018
Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project General Usage	57	4137	May 6, 2024
Apache Arrow 1.0 release Data arrow	7	1968	September 5, 2020
[RFC/ANN] Parquet3.jl — alternative Parquet implementation focused on support for nested columns Package Announcements parquet , arrow	9	505	April 10, 2026

Arrow, Feather, and Parquet

Related topics