ANN: BufIO: New I/O interfaces for Julia

Package Announcements
, ,
7

An update after a few months:

  • The package has new been released on the General Registry - under the new name BufferIO, instead of BufIO as it was originally called, in order to avoid abbreviating too much.
  • The compiler limitation behind Julia issue 53584 has now been lifted, which helps BufferIO performance. This will land in 1.14 if all goes well.
  • With some more minor optimisations, BufferIO now pulls further ahead from Base IO. Some benchmarks are below. Please note again that these benchmarks do not the show the full performance potential of BufferIO, because the benchmarks below, for comparison purposes, have to show a use case where there is a 1:1 API correspondance. The true benefit of BufferIO comes when the use case make Base’s API awkward or even unworkable. One example could be look-ahead reading. Here are some recent questions here on Discourse where BufferIO comes in handy:

Reading benchmark

Version 0.2.4, Julia 1.14-dev

julia> using BufferIO

julia> function test_read(io, v::Vector{UInt8})
           empty!(v)
           for i in 1:100
               push!(v, read(io, UInt8))
           end
           for i in 1:10
              push!(v, 0x01)
              readbytes!(io, @views(v[end-1:end]), 1)
           end
           for i in 1:100
               eof(io) && break
               unsafe_read(io, pointer(v, length(v)), UInt(1))
           end
           close(io)
           return v
       end;

julia> @btime test_read(BufReader(open("src/base.jl")), UInt8[]);
  3.604 μs (15 allocations: 8.87 KiB)

julia> @btime test_read(open("src/base.jl"), UInt8[]);
  7.729 μs (12 allocations: 800 bytes)

julia> @btime test_read(open("src/base.jl"; lock=false), UInt8[]);
  4.457 μs (12 allocations: 800 bytes)

julia> function test_write(io, v)
           for i in 1:100
               write(io, i)
               write(io, view(v, i:i+3))
           end
           s = "abcdefghijklmnopq"
           for i in 1:10
               write(io, view(s, i:i+5))
           end
           for i in 1.0f0:0.1f0:25.0f0
                 write(io, i)
           end
           takestring!(io)
        end;

julia> v = rand(UInt8, 1000);

julia> @btime test_write(IOBuffer(), $v);
  5.033 μs (352 allocations: 10.91 KiB)

julia> @btime test_write(VecWriter(), $v);
  1.497 μs (8 allocations: 7.62 KiB)
13 Likes