# Fastest way to save a large number of DataFrames to disk

**URL:** <https://discourse.julialang.org/t/fastest-way-to-save-a-large-number-of-dataframes-to-disk/114005>\
**Category:** Performance\
**Tags:** dataframes, io, arrow\
**Created:** [May 8, 2024, 2:43pm UTC](https://discourse.julialang.org/t/fastest-way-to-save-a-large-number-of-dataframes-to-disk/114005 "2024-05-08T14:43:04Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![liuyxpp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liuyxpp/32/9870_2.png) [@liuyxpp](https://discourse.julialang.org/u/liuyxpp)\
**Post date:** [May 8, 2024, 3:28pm UTC](https://discourse.julialang.org/t/fastest-way-to-save-a-large-number-of-dataframes-to-disk/114005/2 "2024-05-08T15:28:51Z")

</div>

Inspired by this post ([Writing Arrow files by column](https://discourse.julialang.org/t/writing-arrow-files-by-column/113955)), I found a quite fast way to save a large number of dataframes into a single Arrow file in the following way

```julia
open(Arrow.Writer, "test1.arrow") do writer
    for df in values(df_dict_rand)
        Arrow.write(writer, df)
    end
end

```

It completely skip the combining step. If I want to have a single dataframe, simply loading back the Arrow file, it is blazingly fast.

Edit:

An even better approach is to use `Tables.partitioner` and `Arrow.write` which will will use multiple threads to write multiple record batches simultaneously (e.g. if julia is started with `julia -t 8` or the `JULIA_NUM_THREADS` environment variable is set). (from the Arrow.jl doc)

```julia
parts = Tables.partitioner(values(df_dict_rand))
Arrow.write("test.arrow", parts)

```

---

_[View the full topic](https://discourse.julialang.org/t/fastest-way-to-save-a-large-number-of-dataframes-to-disk/114005)._