# Failing to import (relatively) large CSV file with Julia and VSC

**URL:** <https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350>\
**Category:** Data\
**Tags:** performance, csv, arrow\
**Created:** [March 31, 2024, 1:20pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350 "2024-03-31T13:20:10Z")\
**Posts on this page:** 1\
**Showing post:** 17

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [March 31, 2024, 6:09pm UTC](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350/17 "2024-03-31T18:09:19Z")

</div>

My final suggestion:

First step: Convert the .csv file to .arrow format using the convert.jl script:

```julia
using CSV, Arrow

FILENAME_FULL = "20240110_120secMother_AllCountries_002_T-Results_2022_059_Markup001(full).csv"
OUT_FILE = "20240110_120secMother_AllCountries_002_T-Results_2022_059_Markup001(full).arrow"

Arrow.write(OUT_FILE, CSV.File(FILENAME_FULL; header=false, types=Float32))
nothing

```

Second step: Read the .arrow file and convert it to an array (if that is what you need):

```julia
using Arrow, Tables

IN_FILE = "20240110_120secMother_AllCountries_002_T-Results_2022_059_Markup001(full).arrow"

m = nothing
GC.gc(true)
m = Tables.matrix(Arrow.Table(IN_FILE))
println("Size of matrix variable: $(Base.summarysize(m)/1e9) GB")
nothing

```

The first script needs 45 seconds on my PC (Ryzen 7950X), the second script 4.5s.

@rocco_sprmnt21 used a matrix of 40000x4000 elements, we have:

```julia
julia> m
39360×39360 Matrix{Float32}

```

which is ten times as large…

---

_[View the full topic](https://discourse.julialang.org/t/failing-to-import-relatively-large-csv-file-with-julia-and-vsc/112350)._