Does eachline loads whole stream/file in memory?

General Usage
1

My problem is following. I have a huge .csv file stored in S3 bucket. I don’t want to load it on disk, just read it line by line with the consecutive processing of each line. For the locally stored files I usually use CSV.Rows or for row in CSV.file(). However these options are unavailable in the case of S3 stored data. So, I use following approach:

open(s3path) do stream
        for line in eachline(stream)
            println(line)
        end
    end

My question is, does ```eachline load only one line into memory, so I could read huge .csv files without worrying about OOM problems?

2

I think eachline doesn’t load more than it needs, but I think open(f, ::S3Path) does load the whole thing into memory, which is an issue with AWSS3: Support for streaming `S3Path` data · Issue #204 · JuliaCloud/AWSS3.jl · GitHub.

1 Like