LibDeflate.jl v1.0.0: Fast gzip/zlib de/compression

Package Announcements
1

I’m excited to announce version 1.0.0 of LibDeflate.jl.

What is LibDeflate

LibDeflate.jl wraps Eric Biggers’s libdeflate, which is a fast implementation of gzip, zlib, Adler32, crc32 and DEFLATE.

LibDeflate is significantly faster than CodecZlib. For example, gzip compressing 10 copies of the Bible takes 540 ms for LibDeflate.jl versus 1,748 ms for CodecZlib.jl. The disadvantage is that it does not work with streamed data; you must de/compress entire buffers in memory.
This makes it ideal for when you de/compress many small payloads.

LibDeflate.jl is supposed to be a relatively low-level package which provides a small-ish wrapper around libdeflate. There are fewer high-level convenience functions, but on the flip side the semantics of the functions’ semantics are well-specified.

As a package, LibDeflate.jl as the following features:

  • Completely inferrible and trimmable
  • Most operations do not throw, but return error values for easy and efficient error handling
  • All de/compression methods also work on pointers, and so memory allocated from e.g. a C library can also be de/compressed
  • Allocations are minimized (though not completely eliminated)
  • It currently has a only single dependency: libdeflate_jll

Changes in version 1

Version 1 is the only major changes to LibDeflate since its creation in 2021. It all began when an AI agent found eleven (!) bugs in v0.4, despite the package having 100% code coverage.
Looking at the bugs convinced me they were symptoms of API issues, which caused me to revisit some of the API, and it kind of snowballed from there.

Since I was releasing a breaking change anyway, I revisited every single API, so the list below is not a thorough list of changes. However, some notable changes are:

  • It is now possible to decompress multi-member gzip files
  • More gzip header metadata is now returned: Flags, extra flags and OS
  • There is now a proper documentation page
  • Allocations have been minimized
  • New functions provide upper bounds on compressed data size.
  • The error enum’s values are now documented

While some of the original 11 bugs were fixed in version 0.4.4, I recommend all users move to v1.

2

Love this! @drvi and I had played around with GitHub - JuliaIO/CodecZlibNG.jl: zlib-ng codecs for TranscodingStreams.jl. · GitHub at one point because it was supposed to be faster.

I also started a native Julia port of the go flate library a couple years ago: GitHub - quinnj/Flate.jl · GitHub, but got caught up with some manual assembly necessary. I bet AI could help push that over the finish line these days.

I’ve wanted faster/better zlib for a while, thanks for publishing this!

3

LibDeflate has a bit of a memory management issue right now. For example:

julia> using LibDeflate

julia> for i in 1:10000000
       Compressor(0x09)
       end
Killed

Is killed by OOM.

In CodecZlib.jl I used a custom alloc function to fix this.

It would be great to have a Julia port of libdeflate in GitHub - JuliaIO/ChunkCodecs.jl: A consistent Julia interface for lossless encoding and decoding of bytes in memory · GitHub to speed up JLD2 and Zarr too. Though maybe we need something like RFC: target-specific code · Issue #62193 · JuliaLang/julia · GitHub if manual assembly is required to match performance (IIRC this is needed just for the crc32 parts).

4

I’m not sure I understand. Is it necessarily an issue that you can’t allocate 10 million Compressors at once? Or are you saying the issue is that Julia’s GC doesn’t invoke the finalizers properly to free these objects? And how does your custom allocation function fix this?

FWIW, I can’t reproduce your issue on Julia 1.13 (my Julia seems to GC the Compressors during the loop execution)

Also, why do we necessarily care that it’s a Julia port for ChunkCodecs - what’s the issue with ccall’ing?

5

I made an attempt on that many years ago for some of the decompression functionality. The port was fairly literal but I skipped some prefetch things. The speed was maybe 2-3 times slower than libdeflate, but still faster than Zlib.

6

At least on my machine (Linux, Julia 1.13.0-rc3) julia is not calling the finalizers before the system runs out of memory. The custom allocation function in CodecZlib uses jl_malloc instead of malloc so the julia runtime knows how much memory libzlib is using and can trigger GC more frequently.

Yes the ChunkCodecs interface works with ccall (it is deliberately a lowest common denominator for a number of different C libraries), but I think it would be cool to have pure Julia version, especially for JLD2 which is itself a pure Julia implementation of the HDF5 C library.
JLD2 also wants a slight variant of libdeflate’s decompress! function where the output can be resized if the initial output size isn’t large enough.

7

Is their crc32 implementation faster than Adler’s code in Zlib (currently used in CRC32.jl)? It doesn’t look as hand-optimized at first glance.

8

Yes, LibDeflate’s CRC32 is several times faster than the one in zlib and rivals CRC32c. Here is a small benchmark (on an x86 laptop w/ AVX512):

julia> using LibDeflate, CRC32c, CRC32, BenchmarkTools
       for i in 10:5:30
           println("$(2^i) bytes")
           m = rand(UInt8, 2^i)
           @assert LibDeflate.crc32(m) == CRC32.crc32(m)
           print("\tLibDeflate: "); @btime LibDeflate.crc32($m)
           print("\tCRC32:      "); @btime CRC32.crc32($m)
           print("\tCRC32c:     "); @btime CRC32c.crc32c($m)
       end
1024 bytes
        LibDeflate:   27.146 ns (0 allocations: 0 bytes)
        CRC32:        245.406 ns (0 allocations: 0 bytes)
        CRC32c:       36.559 ns (0 allocations: 0 bytes)
32768 bytes
        LibDeflate:   405.100 ns (0 allocations: 0 bytes)
        CRC32:        5.522 μs (0 allocations: 0 bytes)
        CRC32c:       1.002 μs (0 allocations: 0 bytes)
1048576 bytes
        LibDeflate:   14.067 μs (0 allocations: 0 bytes)
        CRC32:        173.403 μs (0 allocations: 0 bytes)
        CRC32c:       26.560 μs (0 allocations: 0 bytes)
33554432 bytes
        LibDeflate:   568.498 μs (0 allocations: 0 bytes)
        CRC32:        5.809 ms (0 allocations: 0 bytes)
        CRC32c:       1.262 ms (0 allocations: 0 bytes)
1073741824 bytes
        LibDeflate:   23.133 ms (0 allocations: 0 bytes)
        CRC32:        188.797 ms (0 allocations: 0 bytes)
        CRC32c:       42.799 ms (0 allocations: 0 bytes)

You could update CRC32.jl to call libdeflate_jll. The implementation is trivial (just ccall to libdeflate)