Overburden.jl is a small CLI tool that does a narrow task: Given a set of reads mapped to a reference, sub-sample the reads to a smaller set that guarantees some minimal depth at all positions.
This is useful if you have sequenced to a too high depth and want to remove redundant data. In microbial genomics, it’s common to have isolates sequences with depth > 100x, so applying Overburden can often reduce data size by 90%.
Although the bioinformatics work it does it fairly simple and unremarkable (if useful), I am unreasonably excited about it, since it illustrates what modern Julia is capable of. So, putting Overburden’s usefulness in bioinformatics aside and looking at it from a software developer’s perspective, Overburden has the following cool properties:
- Low latency. It runs in ~0.4 seconds on Julia 1.14-alpha1 on my computer on small files despite having 34 transitive dependencies.
- Trimmable. It trims to a ~26 MB relocatable bundle which has a latency of ~15 miliseconds (again, despite all the dependencies!).
- Throughput. The trimmed binary subsamples a 230 MB BAM file to a 35 MB one in ~1.2 seconds with 4 threads on my laptop. Running a comparable[1]
samtoolscommand takes ~1.4 seconds.
It is also an excellent stress test for the stack of bioinfo Julia tools I have authors the last years: BufferIO, BGZFLib, CIGARStrings and LibDeflate. Almost all the performance optimization that makes Overburden this fast has gone into these packages (and Julia itself), not in Overburden directly.
The ability to make these kinds of tools now in Julia is absolutely glorious. Thanks to all those who made it possible: @cshen for building OptParse which powers its CLI, Oscar and Jameson for the Memory abstractions this is built on, Gabriel, Cody and Jeff for working on JuliaC, Kristoffer for making installable Julia apps a possitbility, and no doubt many others.
Development has not been butter smooth yet, however. Some things that would be nice to have support for in Julia, but which we don’t, and therefore I’ve had to hack around are:
- The ability to set the number of threads from within-process, so it can be controlled by the CLI
- The ability to install trimmed apps directly from Pkg so users wouldn’t have to learn how to use JuliaC
- The ability to statically link and/or cross compile binaries to users wouldn’t even have to install Julia to use it
- A trimmable logging package, maybe built on the future TypedCallable type Cody is working on
- Trimmable StyledStrings, so my CLI can have beautiful TUI (@tecosaur is working on it, no doubt)
- Removal of unneeded bundled libraries. I have a build script which
rm’s e.g. OpenBLAS in the bundle since it is not needed, but that is kind of sketchy. Ideally, JuliaC would detect that I never invoke BLAS, and never include it in the first place.
Overburden’s algorithm does two passes over the BAM file: One to pick names and one to subsample based on them and write output. Therefore, I timed
samtools flagstat -@ 3 in.bam > /dev/null && samtools view -@ 3 -b -l 5 -N names.txt -o /tmp/foz2 in.bamto compare: This also does two read passes, one cheap and expensive, and one BAM write. ↩︎
