I’m excited to announce version 1.0.0 of LibDeflate.jl.
What is LibDeflate
LibDeflate.jl wraps Eric Biggers’s libdeflate, which is a fast implementation of gzip, zlib, Adler32, crc32 and DEFLATE.
LibDeflate is significantly faster than CodecZlib. For example, gzip compressing 10 copies of the Bible takes 540 ms for LibDeflate.jl versus 1,748 ms for CodecZlib.jl. The disadvantage is that it does not work with streamed data; you must de/compress entire buffers in memory.
This makes it ideal for when you de/compress many small payloads.
LibDeflate.jl is supposed to be a relatively low-level package which provides a small-ish wrapper around libdeflate. There are fewer high-level convenience functions, but on the flip side the semantics of the functions’ semantics are well-specified.
As a package, LibDeflate.jl as the following features:
- Completely inferrible and trimmable
- Most operations do not throw, but return error values for easy and efficient error handling
- All de/compression methods also work on pointers, and so memory allocated from e.g. a C library can also be de/compressed
- Allocations are minimized (though not completely eliminated)
- It currently has a only single dependency:
libdeflate_jll
Changes in version 1
Version 1 is the only major changes to LibDeflate since its creation in 2021. It all began when an AI agent found eleven (!) bugs in v0.4, despite the package having 100% code coverage.
Looking at the bugs convinced me they were symptoms of API issues, which caused me to revisit some of the API, and it kind of snowballed from there.
Since I was releasing a breaking change anyway, I revisited every single API, so the list below is not a thorough list of changes. However, some notable changes are:
- It is now possible to decompress multi-member gzip files
- More gzip header metadata is now returned: Flags, extra flags and OS
- There is now a proper documentation page
- Allocations have been minimized
- New functions provide upper bounds on compressed data size.
- The error enum’s values are now documented
While some of the original 11 bugs were fixed in version 0.4.4, I recommend all users move to v1.