Read file to io

General Usage
1

Given a filename and an io::IO, what’s the recommended way of reading a file and dumping it to io?

A naive solution is

function read_to_io(filename::AbstractString, io::IO)
    open(filename, "r") do src_io
        while !eof(src_io)
            write(io, read(src_io, UInt8))
        end
    end
end

but I am looking for something efficient. I could use a buffer as in

function read_to_io(filename::AbstractString, io::IO; bufsize = 2^12)
    buffer = Vector{UInt8}(undef, bufsize)
    open(filename, "r") do src_io
        while !eof(src_io)
            n = readbytes!(src_io, buffer, bufsize)
            write(io, @view buffer[1:n])
        end
    end
end

but I am not sure what the buffer size should be. Alternative solutions welcome.

2 Likes
2

The optimal buffer size will depend on the read & write speeds of your disk, as well as whether you’re writing to a disk or the network in the first place. There’s no “one size fits all” here, but chunks in the size of a few megabytes (especially if everything is going off of a PCIe SSD) will likely still be performant.

2 Likes