How to read lines from a file with a reusable buffer?

General Usage
1

What I want to do:

line_buffer = UInt8[]
buffered_io_stream = open("file.txt")

while readuntil!(line_buffer, buffered_io_stream, '\n') != 0
     # do thing with line_buffer
     # clear line_buffer
end

Is there a package out there that does this, ideally into a Vector{UInt8}?

Looking at the source code, readuntil is delegated to some C code? julia/iostream.jl at f9720dc2ebd6cd9e3086365f281e62506444ef37 · JuliaLang/julia · GitHub

Is there any good documentation on the IO / Stream / Buffered versions of those and how to implement them? All I’m seeing is this section of the docs: I/O and Network · The Julia Language Which just describes the methods but doesn’t really get into the expected interfaces for something to be a sub-type of IO for example.

Lastly, are there plans to improve the standard library with regards to Strings/byte strings and improved IO code written in julia instead of C? From what I’m seeing here: Milestones - JuliaLang/julia · GitHub, there are no significant changes planned for Strings or File IO related APIs.

2

I’m actually pretty sure that there is no easy way to do this given what exists in the stdlib. I’d have to implement my own buffered reader since there is no function like “fill buffer” on IOStream.

3

There is read!(::IO, a::Array{UInt8}).

4

True, but that fills an array I pass it. There is no method to fill the IOStream.ios buffer, which would be needed to implement an alternate read_until!.

Here is what I’ve tried:

function readuntil2!(ret::Vector{UInt8}, stream::IOStream, delim::UInt8, keep::Bool=false)
    @assert isreadable(stream)
    resize!(ret, 0)
    buffer1 = stream.ios
    resize!(ret, 0)
    filled = 0
    # Here is where I want to read bytes into the stream.ios buffer, but I don't see any
    # functions that do that
    while read(stream, 4096) > 0
        pos = findfirst(x -> x == delim, buffer1)
        if isnothing(pos)
            sz = length(buffer1)
            if length(ret) < filled + sz
                resize!(ret, filled + sz)
            end
        else
            sz = pos - buffer1.bufferpos + 1
            resize!(ret, filled + sz)
        end
        copyto!(ret, filled + 1, buffer1, sz)
        filled += sz
        if !isnothing(pos)
            break
        end
    end
    return ret
end

# And also
function readuntil3!(ret::Vector{UInt8}, stream::IOStream, delim::UInt8, keep::Bool=false)
    @assert isreadable(stream)
    resize!(ret, 0)
    # This give me a weird error about not being able to determine the return type
    ccall(:jl_readuntil, ret, (Ptr{Cvoid}, UInt8, UInt8, UInt8), stream.ios, delim, 0, !keep)
    nothing
end