Skipping a lot of lines in CSV.read() allocates too much memory

[jules]

@rocco_sprmnt21

Why is mmap() so much faster than read() and seek()

I used mmap after reading this post

File IO Buffers too small?

You can use GitHub - JuliaIO/BufferedStreams.jl: Fast composable IO streams. But indeed, I have also found the performance of IOStream to be unacceptable for many IO tasks where you typically read many small values. One issue with IOStream is that it takes a lock on every IO operation in order to be thread-safe. For binary files, I typically resort to memory mapping the file and wrapping it in an IOBuffer. So instead of io = open("some very large file.dat") I do using Mmap io = IOBuffer(M…