# How to plumb together download -\> uncompress -\> untar without writing full downloaded file

**URL:** <https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668>\
**Category:** New to Julia\
**Created:** [February 18, 2022, 1:54am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668 "2022-02-18T01:54:18Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 18, 2022, 1:54am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/1 "2022-02-18T01:54:19Z")

</div>

I would like to extract a particular file from a large (say dozens of Gb) gzipped tar archive located somewhere else on the internet. I don’t want to actually write that whole archive to local disk, but instead stream the bytes to `/dev/null` with just one specified archive member being skimmed off and written to disk.

So I am already able to do this for small files with a plain `IOBuffer` and `Downloads.download`, `Tar.jl`, and `TranscodedStreams`. However for very large files the `IOBuffer` simply grows too large for my RAM, and when I provide a `maxsize` argument to `IOBuffer` `Tar.extract` simply complains about a premature `EOF`, so I’m assuming the the download filled the buffer and returned.

So I think I could go to some effort using `@async` or maybe `Channels` but I thought I’d ask here first as I have a suspicion that this shouldn’t be that hard with the right sort of `Stream` object.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [February 18, 2022, 2:34am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/2 "2022-02-18T02:34:13Z")

</div>

One idea would be to use an anonymous memory map:  
[https://docs.julialang.org/en/v1/stdlib/Mmap/#Mmap.Anonymous](https://docs.julialang.org/en/v1/stdlib/Mmap/#Mmap.Anonymous)

Technically, this does write some of the data to disk

However, I think may need to access the stream API of zlib.

Currently CodecGzip.jl only exposes some of the API via `ccall` here:  
[https://github.com/JuliaIO/CodecZlib.jl/blob/master/src/libz.jl](https://github.com/JuliaIO/CodecZlib.jl/blob/master/src/libz.jl)

You may need to access functions not currently bound by CodecZlib.jl. In that case, you may need to `ccall` these for yourself referring to the C header:  
[https://github.com/madler/zlib/blob/master/zlib.h](https://github.com/madler/zlib/blob/master/zlib.h)

Perhaps the problem is actually with Tar.jl?

Anyways, perhaps showing us your code would help?

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 18, 2022, 2:59am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/3 "2022-02-18T02:59:51Z")

</div>

Here is simple code that fails. I’m using a local file for testing. We can leave `Gzip` out of it for now for simplicity:

```julia
p = IOBuffer(maxsize=100000)
Download.download("file:./local_archive_for_testing.tar",p)
loc = Tar.extract(x->x.path="file_I_want",p)

```

What I’m asking for is for `p` to work like a pipe between `download` and `Tar.extract`

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 18, 2022, 3:27am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/4 "2022-02-18T03:27:50Z")

</div>

```julia
import Downloads, Tar
io = PipeBuffer()
Downloads.download("file://localhost/path/to/file.tar", io)
loc = Tar.extract(x -> x.path == "file_I_want", io)

```

(but I think this downloads the whole file into the buffer before the `Tar.extract`, which is not what you want)

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 18, 2022, 4:49am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/5 "2022-02-18T04:49:17Z")

</div>

I think you’ll want some `@async` here because you want the decoding to overlap in time with the download. (In principle it’s possible to avoid the `@async` if you implement a custom stream to pass to `download`. This custom stream would need to drive the selection of your desired files out of the archive from within that custom stream’s `write()` method. But this seems fairly messy, unless someone happens to have implemented that already.)

For an async solution, you want a buffered _blocking_ stream instead. There is one such stream in the undocumented `Base.BufferStream` which can be used for this purpose:

```julia
import Downloads, Tar, 
using CodecZlib

# Some necessary piracy - BufferStream doesn't have
# an implementation for `Base.skip()`
function Base.skip(io::Base.BufferStream, n)
    if n > 0
        read(io, n)
    else
        error("Can't skip backward in BufferStream")
    end
end

io = Base.BufferStream()
@sync begin
    @async begin
        Downloads.download("file://localhost/home/chris/tmp/testdata.tgz", io)
        @info "Download complete"
        close(io)
    end
    @async begin
        loc = Tar.extract(x -> x.path == "1.dat" ? (@info("Extracting", x); true) : (@info("Ignoring", x); false), GzipDecompressorStream(io))
        @info "Untar complete" loc
    end
end

```

Testing, we can see that extraction happens in a streaming manner:

```julia
┌ Info: Ignoring
└ x = Tar.Header("10.dat", :file, 0o644, 100000000, "")
┌ Info: Extracting
└ x = Tar.Header("1.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("2.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("3.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("4.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("5.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("6.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("7.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("8.dat", :file, 0o644, 100000000, "")
┌ Info: Ignoring
└ x = Tar.Header("9.dat", :file, 0o644, 100000000, "")
[ Info: Download complete
┌ Info: Untar complete
└ loc = "/tmp/jl_p3kvwB"

```

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 18, 2022, 7:49am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/6 "2022-02-18T07:49:46Z")

</div>

This looks like what I want. So I tried it out on a real internet file but it failed: I got no ‘Ignored’ messages after the first file in the archive, although the download proceeded to completion. It looks like `Tar.extract` ended up somewhere it shouldn’t in the stream, stopped consuming the buffer, and meanwhile `download` kept filling up the buffer and my RAM was starved.

```julia
julia> progress(total,now) = begin
       print("\r$now")
       end
progress (generic function with 1 method)

julia> io = Base.BufferStream()
BufferStream() bytes waiting:0, isopen:true

julia> @sync begin
       @async begin
       Downloads.download("https://data.proteindiffraction.org/ssgcid/3lls.tar",io,progress=progress)
       @info "download complete"
       close(io)
       end
       @async begin
       loc = Tar.extract(x -> x.path == "3lls/series/200873f12_x0181.img.bz2" ? (@info("Extracting",x); true) : (@info("Ignoring", x); false), io)
       @info "Untar complete" loc
       end
       end
┌ Info: Ignoring
└ x = Tar.Header("3lls/", :directory, 0o775, 0, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/", :directory, 0o755, 0, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/200873f12_x0152.img.bz2", :file, 0o664, 1250416, "")
499578880[ Info: download complete
ERROR: TaskFailedException
   nested task error: This does not appear to be a TAR file/stream — invalid version string for tar file: "\x1c>". Note: Tar.jl does not handle decompression; if the tarball is compressed you must use an external command like `gzcat` or package like CodecZlib.jl to decompress it. See the README file for examples.

```

Note that if I download the file to local disk, then run the same `Tar.extract` command, everything works correctly.

```julia
julia> oo = open("3lls.tar","r")
IOStream(<file 3lls.tar>)

julia> loc = Tar.extract(x -> x.path == "3lls/series/200873f12_x0181.img.bz2" ? (@info("Extracting",x); true) : (@info("Ignoring", x); false), oo)
┌ Info: Ignoring
└ x = Tar.Header("3lls/", :directory, 0o775, 0, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/", :directory, 0o755, 0, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/200873f12_x0152.img.bz2", :file, 0o664, 1250416, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/200873f12_x0318.img.bz2", :file, 0o664, 1248919, "")
#truncated
Info: Extracting
└ x = Tar.Header("3lls/series/200873f12_x0181.img.bz2", :file, 0o664, 1246912, "")
┌ Info: Ignoring
└ x = Tar.Header("3lls/series/200873f12_x0056.img.bz2", :file, 0o664, 1246759, "")
#truncated

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 19, 2022, 5:16am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/7 "2022-02-19T05:16:32Z")

</div>

You may need to modify the definition of `skip()` I gave you to ensure exactly `n` bytes are read. Try the following?

```julia
function Base.skip(io::Base.BufferStream, n)
    if n > 0
        read(io, n, all=true)
    else
        error("Can't skip backward in BufferStream")
    end
end

```

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 20, 2022, 1:33pm UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/8 "2022-02-20T13:33:14Z")

</div>

So that didn’t work, it turns out `all` is not a valid option when `io` is a `BufferStream`.

I suspect it is indeed the `skip` method that is causing the problem. The code paths in `stream.jl` for `readbytes!` are different depending on how large the read is.

I’ve also verified that everything works for a large tar file “downloaded” using `file:` protocol from local storage, it just fails when downloading across the network, which sounds like some problem with Tar waiting for the buffer to fill.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 21, 2022, 12:41am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/9 "2022-02-21T00:41:13Z")

</div>

> which sounds like some problem with Tar waiting for the buffer to fill

Yes I thought there’s some short read happening here. If `all` doesn’t work, I was hoping just calling `read` again with the remaining size would be ok, but this doesn’t seem to work either.

The main trouble we’re having here is that `Base.BufferStream` isn’t a public API so it’s not that well tested, some things like `skip` are missing and the precise blocking behavior isn’t really documented. (I noticed something a bit weird about blocking — `BufferStream` seems to only block on `read`, not `write`, so the internal buffer might be able to grow indefinitely. Which is fine in this case because the Tar reader is likely faster than the writer doing the download, but really not great if you were tarring and uploading!)

Another alternative is to use `Pipe`, which is a publicly defined API and widely used for several things. IIUC the cost of blocking on a read or write to `Pipe` will be a lot higher than `BufferStream` because the `Pipe` needs to go through the operating system kernel. But the pipe should have a fixed size buffer, so should block on both read or write side which should be a lot more sensible in general.

The following seemed to work for me:

```julia
using Tar, Downloads

function Base.skip(io::Union{Base.BufferStream,Pipe}, n)
    if n > 0
        while n > 0 && isopen(io)
            buf = read(io, n)
            n -= length(buf)
            #if n > 0
            # @info "Short read" length(buf)
            #end
        end
    else
        error("Can't skip backward in Pipe or BufferStream")
    end
end

io = Pipe()
# Initialize the pipe. I'm not sure there's a public API for this ??
Base.link_pipe!(io)
# Alternatively, use BufferStream... which should work
# but seems to get stuck for some reason
# io = Base.BufferStream()
@sync begin
    @async try
        Downloads.download("https://data.proteindiffraction.org/ssgcid/3lls.tar", io)
        @info "Download complete"
    catch exc
        @error "Caught exception" exc
    finally
        close(io)
    end
    @async try
        loc = Tar.extract(x -> x.path == "3lls/series/200873f12_x0181.img.bz2" ? (@info("Extracting", x); true) : (@info("Ignoring", x); false), io)
        @info "Untar complete" loc
    catch exc
        @error "Caught exception" exc
    finally
        close(io)
    end
end

```

If you wanted to abort the download once the particular file of interest has been read and extracted, you may be able to do `close(io)`.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [February 21, 2022, 1:31am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/10 "2022-02-21T01:31:34Z")

</div>

> [@c42f](#):
>
> The main trouble we’re having here is that `Base.BufferStream` isn’t a public API

There’s an issue for this, by the way: [Document `Base.BufferStream` · Issue #42424 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/42424)

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 21, 2022, 1:36am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/11 "2022-02-21T01:36:52Z")

</div>

Yes, the solution with `Pipe` works, thanks for figuring that out for me! Extra time in the OS is not an issue as download speed will dominate. My diagnosis with `BufferStream` had got to the point where I was pretty sure that the `download` task wasn’t filling the buffer fast enough for the `Tar` task, and instead of waiting the `Tar` task was doing something else. So the same code “downloading” from a local file works - the file read from local disk can keep up with `Tar`.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 21, 2022, 2:30am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/12 "2022-02-21T02:30:06Z")

</div>

> [@frtps](#):
>
> instead of waiting the `Tar` task was doing something else

What should ideally happen is that the tar `Task` should block indefinitely waiting for `read()` to complete with the requested bytes. While it’s blocked, the download `Task` should be running, slowing `write()`ing to the stream. Internally it seems that `Base.readbytes!()` can sometimes return short reads, but I don’t know the situations when this can happen, or whether it’s ever applicable to `BufferStream`. Also the fact that `BufferStream` doesn’t seem to block on `write()` at some max buffer size seems concerning.

Probably @jameson would have some insight into these issues.

---

<div class="post-metadata">

**Author:** ![jameson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jameson/32/23_2.png) [@jameson](https://discourse.julialang.org/u/jameson)\
**Post date:** [February 22, 2022, 3:45pm UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/13 "2022-02-22T15:45:34Z")

</div>

yes, feel free to submit PRs to improve it

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [February 22, 2022, 4:14pm UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/14 "2022-02-22T16:14:09Z")

</div>

See also [https://github.com/staticfloat/SimpleBufferStream.jl](https://github.com/staticfloat/SimpleBufferStream.jl).

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 23, 2022, 3:34am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/15 "2022-02-23T03:34:33Z")

</div>

> [@fredrikekre](#):
>
> See also [GitHub - staticfloat/SimpleBufferStream.jl: What Base.BufferStream should be](https://github.com/staticfloat/SimpleBufferStream.jl)

Nice suggestion, I didn’t know about this package.

@frtps it looks like SimpleBufferStream.jl is ideal and can be used instead of `Pipe()`. And certainly instead of `Base.BufferStream` as it currently exists.

I guess this should eventually replace the Base.BufferStream implementation but for now it’s nice to have in a separate package.

---

<div class="post-metadata">

**Author:** ![frtps](https://avatars.discourse-cdn.com/v4/letter/f/73ab20/32.png) [@frtps](https://discourse.julialang.org/u/frtps)\
**Post date:** [February 23, 2022, 4:17am UTC](https://discourse.julialang.org/t/how-to-plumb-together-download-uncompress-untar-without-writing-full-downloaded-file/76668/16 "2022-02-23T04:17:48Z")

</div>

I’ll try that too then, although `Pipe` works fine. My current problem is that raising an error in the `Tar.extract` predicate function before complete traversal of the tar file (in order to save download time) seems to make the extracted file disappear from the filesystem but that’s probably a new topic.