# ANN: BufIO: New I/O interfaces for Julia

**URL:** <https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347>\
**Category:** Package Announcements\
**Tags:** package, announcement, io\
**Created:** [September 13, 2025, 1:11pm UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347 "2025-09-13T13:11:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jakobnissen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobnissen/32/13477_2.png) [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)\
**Post date:** [September 13, 2025, 1:11pm UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/1 "2025-09-13T13:11:45Z")

</div>

I’m pleased to announce a new package BufIO.jl.

## Overview of BufIO.jl

BufIO provides new and improved I/O interfaces for Julia inspired by Rust, and designed around exposing buffers to users in order to explicitly copy bytes to and from them. Compared to the `Base.IO` interface, the new interfaces in this package are:

- Lower level
- Faster
- Easier to reason about
- Better specified, with more well-defined semantics
- Free from slow fallback methods that silently trash your performance

Beside the new interfaces, BufIO also provides a small set of basic types to make use of the new interface, and/or allow easy interoperation between `Base.IO` types and the new buffered interfaces.

The new types include:

- `BufReader <: AbstractBufReader`: A type that wraps a `Base.IO` to provide the new `AbstractBufReader` interface
- `BufWriter <: AbstractBufWriter`: A type that wraps a `Base.IO` to provide the new `AbstractBufWriter` interface
- `CursorReader <: AbstractBufReader`: Wrap any contiguous memory-backed bytes in a stateful reader
- `IOReader <: Base.IO`: A type that wraps an `AbstractBufReader` and provides the `Base.IO` interface
- `VecWriter <: AbstractBufWriter`: A faster and simpler alternative to `IOBuffer` usable e.g. to build strings.

## Comparison to other packages

The packages BufferedStreams.jl and TranscodingStreams.jl also provide IO wrapper types that buffer their wrapped io. However, both these packages do so as a transparent optimisation, whereas BufIO.jl provides a different interface.

## History of BufIO.jl

I’ve been writing a bunch of different parsers in Julia for about five years. Each time, I’ve found it necessary to create a reader type containing a buffer, then read into the buffer, and then do all my actual parsing on the buffer. From what I hear from others, that’s also how they do it. Somehow, after years of doing that, I didn’t put two and two together about what that implied about the lack of performance and convenience Julia IO interface.

A few years ago, I learned Rust. One of the most common Rust performance footguns is doing IO operations on unbuffered data (unlike Julia, Rust does not buffer several of its basic IO types).  
On the other hand, Rust has the `BufRead` trait, which I’ve found _extremely_ well designed and usable. That led me to believe that an IO interface should be buffered by default, and center its API around the buffer.

The penny finally dropped about a year ago, after a few packages forced me to attempt to handle generic `Base.IO` objects in Julia, which I found to be a miserable experience due to… **many, many** issues of the poorly designed `Base.IO`. I tried to push a (backwards compatible) extension to Base’s IO interface, but it’s difficult to make sweeping changes to Base without close collaboration with someone with commit rights. So:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/a/ea5e84f25dd71c107ace18f3ae9ecc3437e7559f.jpeg)

Jokes aside, BufIO.jl is my vision for an alternative I/O interface in Julia, which I [previously published here](https://hackmd.io/UgeBwtIkTTipkgZSQnXc8w). Ideally, **the interface should make it to Base** , but I’m skeptical that will ever happen. Whether it does or not, it’s useful to prototype the interface in a package to at least be more informed about what a different I/O interface would feel like.

BufIO.jl is not yet in registration, but will be registered soon. I’ve used this interface already in other packages and found it quite expressive and fast.

**I’m very much interested in feedback and suggestions for the interface!** Please open issues on the repo [GitHub - BioJulia/BufferIO.jl: The buffer is the interface](https://github.com/BioJulia/BufIO.jl)

## Notes on the design of the new interface

- BufIO has distinct `AbstractBufReader` and `AbstractBufWriter` types instead of Base’s common `IO` type. I’m not 100% sure which is better, but I’ve found that most use of IO uses _either_ writing or reading, not both, and I’ve found that separating the two interfaces makes implementations cleaner and less bloated. It’s also very easy to work around and create a reader/writer type `T` by e.g. creating a function `writer(::T)::AbstractBufWriter`.

- The types in BufIO is by default **not threadsafe**. Locks are slow, and most programs are single threaded. If users want to protect their resources when used concurrently, they’ll need to use a lock themselves. This tradeoff is no different from any other datastructure, such as a `Dict` which we (correctly) understand in Julia should also not be threadsafe by default.

- The current BufIO performance is limited by a few limitiations of Base Julia:

---

<div class="post-metadata">

**Author:** ![tp2750](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tp2750/32/207806_2.png) [@tp2750](https://discourse.julialang.org/u/tp2750)\
**Post date:** [September 13, 2025, 1:56pm UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/2 "2025-09-13T13:56:16Z")

</div>

This looks really cool!

> [@jakobnissen](#):
>
> - Faster

Do you have some benchmarks showing how much faster?

---

<div class="post-metadata">

**Author:** ![jakobnissen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobnissen/32/13477_2.png) [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)\
**Post date:** [September 13, 2025, 2:14pm UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/3 "2025-09-13T14:14:26Z")

</div>

I haven’t made any, but I can cook up some quickly.  
Here, comparing `IOBuffer` and `VecWriter`:

```julia-repl
julia> function test_write(io)
            for i in 1:100
                write(io, i)
            end
            write(io, [0x01, 0x02, 0x03, 0x04, 0x05])
            write(io, "abcdefghijklmnopq")
            for i in 1.0f0:0.1f0:10.0f0
                 write(io, i)
             end
             io isa IOBuffer ? String(take!(io)) : String(io.vec)
       end

julia> @btime test_write(VecWriter());
  612.495 ns (7 allocations: 4.25 KiB)

julia> @btime test_write(IOBuffer());
  2.487 μs (203 allocations: 6.22 KiB)

```

And here comparing `IOStream` with `BufReader{IOStream}`:

```julia-auto
julia> function test_read(io, v::Vector{UInt8})
           empty!(v)
           for i in 1:100
               push!(v, read(io, UInt8))
           end
           for i in 1:10
              push!(v, 0x01)
              readbytes!(io, @views(v[end-1:end]), 1)
           end
           for i in 1:100
               eof(io) && break
               unsafe_read(io, pointer(v, length(v)), UInt(1))
           end
           close(io)
           return v
       end

julia> @btime test_read(BufReader(open("src/base.jl")), UInt8[]);
  4.039 μs (14 allocations: 8.88 KiB)

julia> @btime test_read(open("src/base.jl"), UInt8[])
  7.604 μs (11 allocations: 816 bytes)

julia> @btime test_read(open("src/base.jl"; lock=false), UInt8[]);
  4.340 μs (11 allocations: 816 bytes)

```

Note that the latter test heavily favors `IOStream` because:

1. `BufReader` is a layer of indirection over `IOStream`, and so requires allocating a superfluous buffer and double-copying. If `IOStream` exposed the `AbstractBufReader` interface directly, it would be faster than either.
2. The greatest performance speed comes when you elect to not use the Base APIs that tend to allocate but instead operate on the buffer directly and use functions like `read_into!`. This makes it hard to do an apples-to-apples comparison (e.g. using `BufferedStreams` in the benchmark above would probably yield the same results as `BufReader`)

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [September 21, 2025, 9:41am UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/4 "2025-09-21T09:41:54Z")

</div>

> [@jakobnissen](#):
>
> That restriction will probably be lifted soon-ish.

What does this mean? Do you expect the fix to happen in the compiler soon, or are you saying you’re trying to work around the issue within BufIO.jl?

> [@jakobnissen](#):
>
> Please open issues on the repo [GitHub - BioJulia/BufIO.jl: Interface for efficient IO in Julia](https://github.com/BioJulia/BufIO.jl)

Have you considered moving the package to the JuliaIO org? Seems more relevant than BioJulia. Although, of course, it’s your choice.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [September 21, 2025, 2:22pm UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/5 "2025-09-21T14:22:40Z")

</div>

> [@nsajko](#):
>
> Have you considered moving the package to the JuliaIO org? Seems more relevant than BioJulia.

Perhaps “Bio” now stands for buffered IO! 😄

---

<div class="post-metadata">

**Author:** ![jakobnissen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobnissen/32/13477_2.png) [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)\
**Post date:** [September 22, 2025, 7:42am UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/6 "2025-09-22T07:42:58Z")

</div>

I hope it’ll be fixed in the compiler for Julia 1.13 or 1.14, so I’ll just write code as if it wasn’t a problem and when wait for it to not be.

Oh yeah, it could be moved to JuliaIO, sure.

---

<div class="post-metadata">

**Author:** ![jakobnissen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobnissen/32/13477_2.png) [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)\
**Post date:** [December 22, 2025, 10:16am UTC](https://discourse.julialang.org/t/ann-bufio-new-i-o-interfaces-for-julia/132347/7 "2025-12-22T10:16:34Z")

</div>

An update after a few months:

- The package has new been released on the General Registry - under the new name BufferIO, instead of BufIO as it was originally called, in order to avoid abbreviating too much.
- The compiler limitation behind [Julia issue 53584](https://github.com/JuliaLang/julia/issues/53584) has now been lifted, which helps BufferIO performance. This will land in 1.14 if all goes well.
- With some more minor optimisations, BufferIO now pulls further ahead from Base IO. Some benchmarks are below. Please note again that these benchmarks do not the show the full performance potential of BufferIO, because the benchmarks below, for comparison purposes, have to show a use case where there is a 1:1 API correspondance. The true benefit of BufferIO comes when the use case make Base’s API awkward or even unworkable. One example could be look-ahead reading. Here are some recent questions here on Discourse where BufferIO comes in handy:
  - [Int -\> Str -\> print(io,\_) without allocations - #7 by jakobnissen](https://discourse.julialang.org/t/int-str-print-io-without-allocations/134626/7)
  - [Nerd Sniping: How can I improve the read in of a human-readable file? - #16 by jakobnissen](https://discourse.julialang.org/t/nerd-sniping-how-can-i-improve-the-read-in-of-a-human-readable-file/134517/16)

### Reading benchmark

Version 0.2.4, Julia 1.14-dev

```julia
julia> using BufferIO

julia> function test_read(io, v::Vector{UInt8})
           empty!(v)
           for i in 1:100
               push!(v, read(io, UInt8))
           end
           for i in 1:10
              push!(v, 0x01)
              readbytes!(io, @views(v[end-1:end]), 1)
           end
           for i in 1:100
               eof(io) && break
               unsafe_read(io, pointer(v, length(v)), UInt(1))
           end
           close(io)
           return v
       end;

julia> @btime test_read(BufReader(open("src/base.jl")), UInt8[]);
  3.604 μs (15 allocations: 8.87 KiB)

julia> @btime test_read(open("src/base.jl"), UInt8[]);
  7.729 μs (12 allocations: 800 bytes)

julia> @btime test_read(open("src/base.jl"; lock=false), UInt8[]);
  4.457 μs (12 allocations: 800 bytes)

```

```julia
julia> function test_write(io, v)
           for i in 1:100
               write(io, i)
               write(io, view(v, i:i+3))
           end
           s = "abcdefghijklmnopq"
           for i in 1:10
               write(io, view(s, i:i+5))
           end
           for i in 1.0f0:0.1f0:25.0f0
                 write(io, i)
           end
           takestring!(io)
        end;

julia> v = rand(UInt8, 1000);

julia> @btime test_write(IOBuffer(), $v);
  5.033 μs (352 allocations: 10.91 KiB)

julia> @btime test_write(VecWriter(), $v);
  1.497 μs (8 allocations: 7.62 KiB)

```