# Skipping a lot of lines in CSV.read() allocates too much memory

**URL:** <https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121>\
**Category:** Performance\
**Tags:** csv, io\
**Created:** [February 12, 2024, 9:52pm UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121 "2024-02-12T21:52:47Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![SteffenPL](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steffenpl/32/206270_2.png) [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)\
**Post date:** [February 13, 2024, 1:29pm UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121/8 "2024-02-13T13:29:10Z")

</div>

If you just search for some solution, then the following works

```julia
using CSV, DataFrames

rows = eachline("test.csv")
m = 2
dfs = DataFrame[]

while !Iterators.isempty(rows)
    chunck = IOBuffer(join(Iterators.take(rows, m), "\n"))
    df = CSV.read(chunck, DataFrame; buffer_in_memory=true)
    push!(dfs, df)
end

```

Although it might be terribly slow due to lack of parallelism CSV.jl.

* * *

PS: Do you have control over the input CSV files? It feels much easier to split the files once in several files instead of dealing with such non-standard CSV files.

---

_[View the full topic](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121)._