[ANN] TranscodingStreams.jl - new APIs to zlib, bzip2, xz, zstd and more!

Hi all,

I’d like to introduce my new packages: TranscodingStreams.jl and its friends.

TranscodingStreams.jl is a package to make it simpler and more consistent to read/write encoded data stream. For example, if you use Libz.jl (I am a maintainer of it) TranscodingStreams.jl and CodecZlib.jl will be a drop-in replacement of it. Moreover, CodecBzip2.jl, CodecXz.jl, and CodecZstd.jl support interfaces of TranscodingStremas.jl and hence you can use these all lossless compression algorithms without knowing the details.

Here is an example of reading a gzip-compressed file:

using CodecZlib
stream = GzipDecompressionStream(open("data.txt.gz"))
for line in eachline(stream)
    # do something...

If you want to read a bzip2-, xz-, or zstd-compressrd file instead of gzip in the example above, you can replace GzipDecompressionStream with Bzip2DecompressionStream, XzDecompressionStream, or ZstdDecompressionStream, respectively. These are all aliases to TranscodingStream{C,S} where C<:Codec and S<:IO, which is a subtype of IO and supports a range of I/O operations. More detailed explanation and examples can be found at the docs of TranscodingStreams.jl.

These packages are still quite young and haven’t support Windows yet. I really appreciate pull requests to support Windows platforms if you have access to Windows. I hope to get feedbacks from users.

Thank you.


These look really useful, like a libarchive suite in Julia. And a small amount of code for each one, too.

A post was split to a new topic: GzipDecompressionStream compared to GZip.jl?