# Skipping a lot of lines in CSV.read() allocates too much memory

**URL:** <https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121>\
**Category:** Performance\
**Tags:** csv, io\
**Created:** [February 12, 2024, 9:52pm UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121 "2024-02-12T21:52:47Z")\
**Posts on this page:** 1\
**Showing post:** 34

<div class="post-metadata">

**Author:** ![aris](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@aris](https://discourse.julialang.org/u/aris)\
**Post date:** [February 15, 2024, 11:20am UTC](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121/34 "2024-02-15T11:20:55Z")

</div>

Update:  
I reckon this is the fastest solution I found so far;

```julia
using CSV, DataFrames

number_of_lines = 10^6
CSV.write("data.csv", DataFrame(rand(number_of_lines, 10), :auto))

steps = 10
chunk = round(Int, number_of_lines / steps)

# Preallocate arrays
raw_strings::Vector{String} = [" " for _ in 1:chunk]

open("data.csv") do io
    readline(io) #skip header

    for _ in 1:steps

        @time for i in 1:chunk
            raw_strings[i] = readline(io)
        end

        # split strings and convert to numbers or dataframe
        # do whatever you want with the numbers 
        # move to next block

    end
end

```

Again, thank you everyone for your inputs.

---

_[View the full topic](https://discourse.julialang.org/t/skipping-a-lot-of-lines-in-csv-read-allocates-too-much-memory/110121)._