Continuing the discussion from JDF - an experimental DataFrame serialization format is ready for beta testing : @ExpandingMan Thank you very much! That’s a great summary and it helps me getting idea of reliability level of feather format. It’s also nice to get a summary of the recent activities in A…

[image] xiaodai: The post technically is talking about Arrow (which I think is closely related to parquet). I found the situation quite confusing myself and believe I have pieced it together over time. My understanding is that the original goal of arrow was to provide a common in-memory data…

We have some facilities for serializing and memory-mapping ndarrays using common memory abstractions and metadata serialization in the Arrow project but generic storage or scientific ndarray data hasn’t been a central focus of the community relative to the tabular data problem.

Edit: for those interested there’s some more comments here https://github.com/apache/arrow/issues/4802 Hm. So reading the documentation and your comment: I can see there’s tooling for working with tensors (ndarrays), and I can see the serialization. So what’s missing is a tensor-compatible interfa…

Parquet doesn’t have a tensor/ndarray value type, but you could embed tensor data in a BYTE_ARRAY value if you wanted. The format is not designed for general storage of numerical datasets, for that you would be better off using HDF5. Parquet is designed for analytics / SQL-style query processing

@wesm , could you comment some on feather vs the arrow on disk serialization format? It had seemed to me that feather was developed very early in arrow development, and I suspected this had something to do with why its metadata is completely different from arrows IPC metadata format. It seemed to m…

Can someone explain, what’s the difference between Arrow/Feather/Parquet vs HDF5? It seems the latter is pretty powerful and has a nice performance, except for parallel writes.

@sairus7 comparing HDF5 with a columnar file format is truly apples and oranges. HDF5 does not provide a columnar data model for datasets nor metadata describing schemas for such. I’ve used HDF5 for more than 10 years now, and it serves as “a place to put raw bytes or a collection of ndarrays/tensor…

Thanks for clarifying that, it’s really interesting! Some time ago we have moved our structured storage to HDF5 from a directory tree witn file streams, just because HDF5 compiles for different architectures, has support for streaming data with chunking and compression, and also it’s easy to use fr…

Parquet doesn’t have a tensor/ndarray value type, but you could embed tensor data in a BYTE_ARRAY value if you wanted. The format is not designed for general storage of numerical datasets, for that you would be better off using HDF5. Parquet is designed for analytics / SQL-style query processing I…

The most promising format I have seen so far is probably TileDB. With honorable mention going to the potential future prospects of N5/Zarr (both projects are merging going forward). I think TileDB storage with (optionally) Arrow in-memory objects could do some nifty things :slight_smile:

I was aware of N5/Zarr from a blog post about the pangeo project from the dask blog but I didn’t know about TileDB.That seems pretty sweet. For older array db systems check out http://rasdaman.com/ & https://www.paradigm4.com/try_scidb/

Arrow, Feather, and Parquet

Specific Domains Data

tkf December 2, 2019, 8:49pm 40

For people interested in TileDB, there is a vote for Julia support:

Topic		Replies	Views
Writing Parquet files General Usage	28	5569	November 12, 2020
What fileformat to use to load data for high performance computing Machine Learning	37	7275	December 1, 2018
Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project General Usage	57	4251	May 6, 2024
Apache Arrow 1.0 release Data arrow	7	1985	September 5, 2020
[RFC/ANN] Parquet3.jl — alternative Parquet implementation focused on support for nested columns Package Announcements parquet , arrow	9	535	April 10, 2026

Arrow, Feather, and Parquet

Related topics