# Benchmarking ways to write/load DataFrames IndexedTables to disk

**URL:** <https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973>\
**Category:** Data\
**Created:** [February 10, 2018, 1:13pm UTC](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973 "2018-02-10T13:13:55Z")\
**Posts on this page:** 3\
**Page:** 3

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 24, 2018, 1:22pm UTC](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973/41 "2018-10-24T13:22:16Z")

</div>

As of the last couple of releases, Arrow.jl _only_ works on Julia 1.0, so no worries there.

Note that this package was built primarily with Feather in mind, so we are still missing all the arrow IPC stuff (it’s really just arrays at the moment). As far as I know, at the moment Feather.jl is the fastest way to read and write tabular data from disk in pure Julia. Strings are known to be a _little_ slow, but they are still quite fast. Also note that Feather.jl does lazy loading by default now. Unfortunately we are still limited to 4GB per feather file because the feather standard badly needs to be upgraded, but stay tuned for that.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 25, 2018, 1:51am UTC](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973/42 "2018-10-25T01:51:40Z")

</div>

> [@ExpandingMan](#):
>
> was built primarily with Feather in mind

What about Parquet? It seems more promising to be as it allows compression of the files.

> [@ExpandingMan](#):
>
> Unfortunately we are still limited to 4GB per feather file

If my memory serves me correctly, I don’t think this is true anymore.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 25, 2018, 3:55pm UTC](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973/43 "2018-10-25T15:55:16Z")

</div>

> [@xiaodai](#):
>
> What about Parquet? It seems more promising to be as it allows compression of the files.

It depends on what you are doing. Parquet is definitely a more capable format than Feather, but it is also quite a bit more complicated. I think the intended use case for Feather was just “well, I have a few GB of data that I’m messing around with, I need to store it while I’m working on it for the next few weeks” whereas Parquet was intended as a data source for “production” processes.

To answer your question yes, you should be able to use the arrays provided by Arrow.jl to wrap arrays that appear in Parquet. I have yet to take a serious look at the parquet metadata, so I don’t really know what would be involved in doing that. I’m pretty confident that it would be significantly more complicated than Feather.

> [@xiaodai](#):
>
> If my memory serves me correctly, I don’t think this is true anymore.

If there was any change to the standard, it doesn’t seem to be reflected [here](https://github.com/wesm/feather/blob/master/doc/FORMAT.md). Like I said, I have a number of thoughts on how we should go about this sort of thing, so I may have some news in the future.

[Previous page](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973.md?page=2)
