# Skipping a lot of lines in CSV.read() allocates too much memory

**URL:** <https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121>\
**Category:** Performance\
**Tags:** csv, io\
**Created:** [February 12, 2024, 9:52pm UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121 "2024-02-12T21:52:47Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![aris](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@aris](https://discourse.julialang.org/u/aris)\
**Post date:** [February 12, 2024, 9:52pm UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121/1 "2024-02-12T21:52:47Z")

</div>

Hello everyone,

Consider the following example:

```julia
using CSV, DataFrames

number_of_lines = 10^4
CSV.write("data.csv", DataFrame(rand(number_of_lines,10), :auto))

steps = 20
chunk = round(Int,number_of_lines/steps)

for i in 1:steps
    @time a=CSV.read("data.csv",DataFrame,skipto=i*chunk ,limit=chunk )    
end

```

Effectively, we just split the csv file into “chunks” of rows, and read them one by one.

It seems that when more lines are skipped, a lot more memory is allocated, and thus for reading very big files in chunks (e.g the example above using 10^6 lines), getting towards the end of the file becomes painfully slow.

Is there a way to deal with this issue?  
Would it be more appropriate to use another function to import the data perhaps?

---

_[View the full topic](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121)._