# Export very large dataframe

**URL:** <https://discourse.julialang.org/t/export-very-large-dataframe/69229>\
**Category:** General Usage\
**Tags:** question, dataframes, csv, io\
**Created:** [October 5, 2021, 10:53am UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229 "2021-10-05T10:53:15Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 5, 2021, 7:14pm UTC](https://discourse.julialang.org/t/export-very-large-dataframe/69229/7 "2021-10-05T19:14:09Z")

</div>

> [@bojusemo](#):
>
> I do not see any `compress` keyword argument in the documentation and `compress = true` produces an error. Do you have more information?

Compression is supported in [CSV.jl v.0.9](https://discourse.julialang.org/t/ann-csv-jl-0-9-release/67844).

CSV.jl documentation covers this in the [writing section](https://csv.juliadata.org/stable/writing.html#Writing).

Tested code below in Win10 Julia 1.7 for a dataframe with 200 million rows and 4 columns with Float64.

It took 2-3 min and 15 GB of disk space, in standard csv writing mode (with no compression).

However, compression seems to be very slow for this type of large size random input data. Same slowness using 7Zip, for instance. Compression seems to take \>20 min to complete (aborted it as no patience), and it might achieve only ~50% compression. Tbc.

**NB:** _for smaller random data dataframe, CSV.jl gzip achieved a bit better than 50% compression._

```julia
using CSV, DataFrames

nr = 200_000_000; nc = 4
df = DataFrame(rand(nr,nc), :auto)
CSV.write("df_200M_x_4.csv", df) #no compression: 2-3 min, 15 GB file
CSV.write("df_200M_x_4.gzip", df, compress=true) # > 20 min, ~50% compression? Tbc

```

---

_[View the full topic](https://discourse.julialang.org/t/export-very-large-dataframe/69229)._