# Export very large dataframe

**URL:** <https://discourse.julialang.org/t/export-very-large-dataframe/69229>\
**Category:** General Usage\
**Tags:** question, dataframes, csv, io\
**Created:** [October 5, 2021, 10:53am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229 "2021-10-05T10:53:15Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 5, 2021, 10:53am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/1 "2021-10-05T10:53:15Z")

</div>

Hi guys,

May someone tell me please how to export a 6G dataframe with 200 million rows and 4 columns? I am using the `CSV.write` function of the `CSV` package, but it just exports part of the table. I think it is because the dataframe is too big. Any ideas on how to deal with it? I guess that what I need to do is open a file and write on it, but I do not know how to do it with a dataframe. I really appreciate any help.

Thank you

Boris

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 5, 2021, 10:57am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/2 "2021-10-05T10:57:46Z")

</div>

Have you tried CSV.write `compress` keyword argument?

---

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 5, 2021, 12:23pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/3 "2021-10-05T12:23:56Z")

</div>

> [@rafael.guerra](#):
>
> CSV.write `compress`

Hi @rafael.guerra,  
Thank you for your reply. I do not see any `compress` keyword argument in the documentation and `compress = true` produces an error. Do you have more information?

---

<div class="post-metadata">

**Author:** ![kobusherbst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kobusherbst/32/19682_2.png) [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Post date:** [October 5, 2021, 12:28pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/4 "2021-10-05T12:28:51Z")

</div>

I use Arrow.jl to store very large DataFrames (500 million+) rows. The Arrow record batches feature is also pretty useful in processing very large data sets.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 5, 2021, 12:51pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/5 "2021-10-05T12:51:21Z")

</div>

> [@bojusemo](#):
>
> Any ideas on how to deal with it?

If you are not concerned about stability across julia versions you can try

```julia
using JDF
JDF.save("path/to/file.jdf", df)

# to load it back
df_copy = JDF.load("path/to/file.jdf")

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 5, 2021, 1:05pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/6 "2021-10-05T13:05:03Z")

</div>

> [@bojusemo](#):
>
> but it just exports part of the table

This is a bug, I’m sure CSV is written to export the whole thing but maybe is encountering something that makes it abort.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 5, 2021, 7:14pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/7 "2021-10-05T19:14:09Z")

</div>

> [@bojusemo](#):
>
> I do not see any `compress` keyword argument in the documentation and `compress = true` produces an error. Do you have more information?

Compression is supported in [CSV.jl v.0.9](https://discourse.julialang.org/t/ann-csv-jl-0-9-release/67844).

CSV.jl documentation covers this in the [writing section](https://csv.juliadata.org/stable/writing.html#Writing).

Tested code below in Win10 Julia 1.7 for a dataframe with 200 million rows and 4 columns with Float64.

It took 2-3 min and 15 GB of disk space, in standard csv writing mode (with no compression).

However, compression seems to be very slow for this type of large size random input data. Same slowness using 7Zip, for instance. Compression seems to take \>20 min to complete (aborted it as no patience), and it might achieve only ~50% compression. Tbc.

**NB:** _for smaller random data dataframe, CSV.jl gzip achieved a bit better than 50% compression._

```julia
using CSV, DataFrames

nr = 200_000_000; nc = 4
df = DataFrame(rand(nr,nc), :auto)
CSV.write("df_200M_x_4.csv", df) #no compression: 2-3 min, 15 GB file
CSV.write("df_200M_x_4.gzip", df, compress=true) # > 20 min, ~50% compression? Tbc

```

---

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 10, 2021, 8:32am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/8 "2021-10-10T08:32:34Z")

</div>

Thank you @rafael.guerra,  
I updated Julia to 1.7 and CSV package. After that, a much larger file was exported, but it still did not have all the rows. Maybe there is a limitation resources as I am working in a laptop?

---

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 10, 2021, 8:33am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/9 "2021-10-10T08:33:06Z")

</div>

thanks for your reply @dlakelan

---

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 10, 2021, 8:33am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/10 "2021-10-10T08:33:51Z")

</div>

thanks for your reply @kobusherbst

---

<div class="post-metadata">

**Author:** ![bojusemo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bojusemo/32/26969_2.png) [@bojusemo](https://discourse.julialang.org/u/bojusemo)\
**Post date:** [October 10, 2021, 8:34am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/11 "2021-10-10T08:34:17Z")

</div>

thanks for your reply @xiaodai