# Failing to import (relatively) large CSV file with Julia and VSC

**URL:** <https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350>\
**Category:** Data\
**Tags:** performance, csv, arrow\
**Created:** [March 31, 2024, 1:20pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350 "2024-03-31T13:20:10Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 31, 2024, 10:28pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/22 "2024-03-31T22:28:20Z")

</div>

> [@ufechner7](#):
>
> for a .csv file of 5.9 GB size.

The 4 min were to read a CSV file with a 40\_000 x 40\_000 matrix of floats (30 GB on disk, 6.4 GB on RAM).

Defining the type as Float64 in CSV.File didn’t help in my case.

I will try reading to a matrix tomorrow, instead of a data frame.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [March 31, 2024, 10:31pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/23 "2024-03-31T22:31:44Z")

</div>

> [@rafael.guerra](#):
>
> 30 GB on disk

How large such an array is in .csv format depends on the number of digits you save, and on the number of zeros.

Did you start Julia with `julia -t auto`?

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [April 1, 2024, 8:43am UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/24 "2024-04-01T08:43:02Z")

</div>

Thanks Uwe,  
I was wrong about the conversion time from CSV to Arrow.  
For the record this is what I got for the 30 GB CSV file (on disk) containing a 40\_000 x 40\_000 square matrix of Float64 (6.4 GB on RAM):

```julia
- CSV.File to Dataframe: 3.5 min
- CSV.File to Tables.matrix: 3.2 min
- CSV to arrow conversion: 4.9 min
- Arrow to Tables.matrix 10 s

```

---

<div class="post-metadata">

**Author:** ![etiennebert](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/etiennebert/32/208177_2.png) [@etiennebert](https://discourse.julialang.org/u/etiennebert)\
**Post date:** [April 1, 2024, 10:40am UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/25 "2024-04-01T10:40:26Z")

</div>

Many thanks Uwe for your help.  
It works perfectly now. The arrow conversion is the easiest and quickest for me.  
Thank you again!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [September 22, 2024, 12:53pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/26 "2024-09-22T12:53:54Z")

</div>

This solution is totally rad

[Previous page](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350.md?page=1)