# Read Vector{UInt8} lines from a gzipped file (optimization)

**URL:** <https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135>\
**Category:** General Usage\
**Tags:** question\
**Created:** [September 28, 2017, 10:14am UTC](https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135 "2017-09-28T10:14:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 28, 2017, 10:14am UTC](https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135/1 "2017-09-28T10:14:54Z")

</div>

I am reading (and parsing) a huge dataset, essentially ASCII lines in gzipped files. I have optimized the parsing part, now I am trying to figure out how to just read the lines efficiently.

I am using the excellent `CodecZlib` (which believe is the fastest solution). Extracting the data from the string returned by `readline` is pretty fast, and so is `readuntil` (actually better). I thought I would make it faster by reading into a buffer (line length can be bounded). But it is actually slower.

If anyone wants to help out with making this faster, I made a self-contained MWE, available below. This should offer sub-second runtimes for reading on a recent computer, my actual runtimes are around 30 minutes for the whole dataset, and I do this repeatedly, so every little bit helps.

[https://gist.github.com/tpapp/66f053bcab3f763a72318a9e8ef6e177](https://gist.github.com/tpapp/66f053bcab3f763a72318a9e8ef6e177)

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 28, 2017, 11:55am UTC](https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135/2 "2017-09-28T11:55:16Z")

</div>

Basically, the only thing you want to do is to be able to give a preallocated buffer to `readuntil`. Since TranscodingStreams doesn’t seem to provide this we can hack our own (`readuntil2`) just to try out the performance:

```julia
@eval TranscodingStreams begin
    readuntil2(stream::TranscodingStream, delim::UInt8) = readuntil2!(UInt8[], stream, delim)

    function readuntil2!(ret::Vector{UInt8}, stream::TranscodingStream, delim::UInt8)
        changestate!(stream, :read)
        buffer1 = stream.state.buffer1
        resize!(ret, 0)
        filled = 0
        while !eof(stream)
            pos = findbyte(buffer1, delim)
            if pos == 0
                sz = buffersize(buffer1)
                if length(ret) < filled + sz
                    resize!(ret, filled + sz)
                end
            else
                sz = pos - buffer1.bufferpos + 1
                resize!(ret, filled + sz)
            end
            readdata!(buffer1, ret, filled+1, sz)
            filled += sz
            if pos > 0
                break
            end
        end
        return ret
    end
end

```

We then get:

```julia
julia> struct Buffer
           buffer::Vector{UInt8}
           Buffer() = new(UInt8[])
       end

julia> (buff::Buffer)(io::IO) = TranscodingStreams.readuntil2!(buff.buffer, io, UInt8('\n'))

julia> io = GzipDecompressionStream(open(filename));

julia> buff = Buffer()
Buffer(UInt8[])

julia> @time dolines(buff, io, 1_000_000)
  0.298013 seconds (6.58 k allocations: 154.266 KiB)
53847323

```

Compared to e.g. `readline1`

```julia
julia> io = GzipDecompressionStream(open(filename));

julia> @time dolines(readline1, io, 1_000_000)
  0.555570 seconds (5.01 M allocations: 381.854 MiB, 18.27% gc time)
53847323

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 1, 2017, 1:35pm UTC](https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135/3 "2017-10-01T13:35:56Z")

</div>

@kristoffer.carlsson: Thanks, this is indeed a large improvement (actually 3x faster in some scenarios I am testing).

Just checking: OK if I submit a PR based on your code to `TranscodingStreams`? Which is under MIT “Expat” License.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 1, 2017, 1:58pm UTC](https://discourse.julialang.org/t/read-vector-uint8-lines-from-a-gzipped-file-optimization/6135/4 "2017-10-01T13:58:53Z")

</div>

Sure