Skipping a lot of lines in CSV.read() allocates too much memory

General Usage Performance
,
46

You can do it like this

using CSV, DataFrames

open("test.data") do f
    readuntil(f, "ITEM: ")  # skip over first ITEM: string
    while !eof(f)

        boxsize = CSV.read(IOBuffer(readuntil(f, "ITEM: ")), DataFrame)
        positions = CSV.read(IOBuffer(readuntil(f, "ITEM: ")), DataFrame)

    end
end

You also get the correct labels from the file format, e.g. you can do stuff like positions.xs etc.

3 Likes