# Avoiding small strings and using IOBuffer

**URL:** <https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980>\
**Category:** Performance\
**Tags:** strings, io\
**Created:** [December 1, 2023, 8:07am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980 "2023-12-01T08:07:02Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Post date:** [December 1, 2023, 8:07am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/1 "2023-12-01T08:07:02Z")

</div>

Recently I’ve been using this bit of code to basically conditionally concatenate a bunch of lines in a file:

```julia
buf = IOBuffer()
open("input.txt", "r") do f
    for line in readlines(f)
        if line[1] != "#"
            println(buf, line)
        end
    end
end
out = String(take!(buf))

```

I’m itching to remove that line variable, whether by seeking until I find a hashtag or similar, but I can’t seem to find any way to read in from a file directly into an IOBuffer. There always seems to be a String return.

Am I overlooking something under the hood that means I shouldn’t care about this? Or am I overlooking something really obvious having to do with the fact that the file itself is already an IO object. It really feels like I should be using that somehow…

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [December 1, 2023, 8:20am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/2 "2023-12-01T08:20:16Z")

</div>

> [@grahamas](#):
>
> but I can’t seem to find any way to read in from a file directly into an IOBuffer.

Does

```julia
IOBuffer(read("input.txt"))

```

do what you want?

You can also make the code you’ve written somewhat shorter and more efficient with

```julia
for line in eachline("input.txt")
    if line[1] != "#"
        println(buf, line)
    end
end

```

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [December 1, 2023, 8:33am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/3 "2023-12-01T08:33:08Z")

</div>

I don’t know how it matches your real needs, but you can also get your end result with a one-liner:

```julia
out = join((line for line in eachline(filename, keep = true) if !startswith(line, "#")))

```

---

<div class="post-metadata">

**Author:** ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Post date:** [December 1, 2023, 9:12am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/4 "2023-12-01T09:12:20Z")

</div>

@GunnarFarneback That’s a nice one-liner! But unfortunately none of that avoids the problem of allocating the `line` string, and seems to introduce some overhead besides.

To do some tests, I created a file of random strings on 100,000 lines

```julia
open("testout.txt", "w") do f
    for i in range(1,10000)
        println(f, randstring(5))
    end
end

function test_eachline()
    buf = IOBuffer()
    for line in eachline("testout.txt")
        if line[1] == 'K'
            println(buf, line)
        end
    end
    String(take!(buf))
end

function test_eachline_IOBuffer()
    buf = IOBuffer()
    for line in eachline(IOBuffer(read("testout.txt")))
        if line[1] == 'K'
            println(buf, line)
        end
    end
    String(take!(buf))
end

function test_oneliner()
    join((line for line in eachline("testout.txt", keep = true) if startswith(line, "K")))
end

function test_seekpeek()
    buf = IOBuffer()
    open("testout.txt", "r") do f
        while !eof(f)
            if peek(f, Char) == 'K'
                print(buf, readuntil(f, '\n', keep=true))
            else
                skipchars(!=('\n'), f)
                skip(f, 1)
            end
        end
    end
    String(take!(buf))
end

function test_seekpeek_IOBuffer()
    buf = IOBuffer()
    f = IOBuffer(read("testout.txt"))
    while !eof(f)
        if peek(f, Char) == 'K'
            print(buf, readuntil(f, '\n', keep=true))
        else
            skipchars(!=('\n'), f)
            skip(f, 1)
        end
    end
    String(take!(buf))
end

function test_seekpeek_eachchar()
    buf = IOBuffer()
    f = IOBuffer(read("testout.txt"))
    while !eof(f)
        if peek(f, Char) == 'K'
            ch = read(f, Char)
            while ch != '\n'
                write(buf, ch)
                eof(f) && break
                ch = read(f, Char)
            end
            !eof(f) && write(buf, ch)
        else
            skipchars(!=('\n'), f)
            skip(f, 1)
        end
    end
    String(take!(buf))
end

```

```julia
@btime test_eachline()
# 3.258 ms (100033 allocations: 2.32 MiB)
@btime test_eachline_IOBuffer()
# 3.736 ms (200024 allocations: 15.10 MiB)
@btime test_oneliner()
# 3.604 ms (200035 allocations: 3.85 MiB)
@btime test_seekpeek()
# 226.700 ms (3478 allocations: 312.97 KiB)
@btime test_seekpeek_IOBuffer()
# 2.644 ms (3480 allocations: 872.14 KiB)
@btime test_seekpeek_eachchar()
# 2.648 ms (24 allocations: 615.64 KiB)

```

The last one finally avoided the sheer number of allocations I wanted to avoid (by avoiding allocating `line` on every loop), but maybe proved that there’s something under the hood going on that means I shouldn’t care about all those allocations: the total allocated space actually increased over the seek-peek using `readuntil`.

(edited to hide my use of `@time` and the irrelevant numbers that generated; and again to fix one-liner)

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [December 1, 2023, 9:19am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/5 "2023-12-01T09:19:13Z")

</div>

> [@grahamas](#):
>
> Oh, and the one-liner is weirdly in the “always recompiles” club.

Most likely because it makes a new generator each time you run it. If you place the code inside a function it won’t, and that’s generally a good idea when you benchmark code, as is using the BenchmarkTools package.

---

<div class="post-metadata">

**Author:** ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Post date:** [December 1, 2023, 9:36am UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/7 "2023-12-01T09:36:24Z")

</div>

> [@GunnarFarneback](#):
>
> Most likely because it makes a new generator each time you run it. If you place the code inside a function it won’t, and that’s generally a good idea when you benchmark code, as is using the BenchmarkTools package.

Ah whoops, I should really know better by now; I thought that was just a matter of running it twice. Didn’t realize the generated code could cause problems even after the second time.

I’ve edited the post since I don’t think anyone will actually learn anything from my mis-use of `@time`.

at least now the char-by-char reading generates the fastest code, barely, but it seems the allocations really have nothing to do with it.

edit: sorry, it’s 3am here so I’m editing and deleting all over the place.

edit2: though discourse’s choice to quote my deleted post was definitely not my exhaustion’s fault…

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 1, 2023, 12:47pm UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/8 "2023-12-01T12:47:12Z")

</div>

> [@grahamas](#):
>
> but I can’t seem to find any way to read in from a file directly into an IOBuffer.

In Julia 1.11 there will be a `copyuntil` function that supports this ([julia#48273](https://github.com/JuliaLang/julia/pull/48273)). Until then, you could also use the `eachlineV` function in the [ViewReader](https://github.com/rickbeeloo/ViewReader) package to read lines into a pre-allocated buffer.

Another option would be to `mmap` the array into an array of bytes, construct a `StringView` from it (via StringViews.jl), and then use `eachsplit` to iterate over (copy-free) views of the lines, ala:

```julia
using StringViews, Mmap
buf = IOBuffer()
open("input.txt", "r") do f
    s = StringView(mmap(f))
    for line in eachsplit(s, '\n')
        if !startswith(s, "#")
            println(buf, line)
        end
    end
end
out = String(take!(buf))

```

(Obviously, don’t do this in global scope if you care about performance — [put it into a function](https://docs.julialang.org/en/v1/manual/performance-tips/#Performance-critical-code-should-be-inside-a-function).)

PS. Note that in your original code, `line[1] != "#"` will always be `false` because you are comparing a character `line[1]` to a string `"#"`. It will also throw an error if the line is empty. Use `startswith(line, "#")` or `!isempty(line) && line[1] != '#'`.

---

<div class="post-metadata">

**Author:** ![grahamas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grahamas/32/3401_2.png) [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Post date:** [December 1, 2023, 7:06pm UTC](https://discourse.julialang.org/t/avoiding-small-strings-and-using-iobuffer/106980/9 "2023-12-01T19:06:23Z")

</div>

Fantastic, thank you! [ViewReader](https://github.com/rickbeeloo/ViewReader) is exactly what I was looking for and really takes the cake for speed:

```julia
using StringViews, Mmap
function test_StringViews()
    buf = IOBuffer()
    open("testout.txt", "r") do f
        s = StringView(mmap(f))
        for line in eachsplit(s, '\n')
            if startswith(line, "K")
                println(buf, line)
            end
        end
    end
    String(take!(buf))
end

using ViewReader
function test_ViewReader()
    buf = IOBuffer()
    for line in eachlineV("testout.txt")
        if startswith(line, "K")
            println(buf, line)
        end
    end
    String(take!(buf))
end

```

```julia
@btime test_StringViews()
# 2.416 ms (3407 allocations: 161.67 KiB)
@btime test_ViewReader()
# 776.402 μs (3405 allocations: 234.02 KiB)

```