I’m pleased to announce my package ScanByte.jl. The package does one simple thing: Given an array of bytes to search, and a set of bytes to search for, finds the first occurence of any byte in the set.
The package is useful because it is micro-optimized for speed. On my laptop, it can scan around 20 GB/s, which is the speed at which my RAM can feed by CPU.
Very fast byte search is used for things like Regex implementations, parsers, or other software than searches through text. E.g. ripgrep uses a technique similar to this package.
Even though it’s well tested, the package is probably pretty rough around the corners at this stage. In particular, I haven’t yet settled on the API. Feedback is much appreciated!
There are a few drawbacks/limitations to this package:
-
The byte set to search for must be known at compile time
-
A new specialized function is generated per new unique byte set
-
The package requires a CPU with either the AVX2 or the SSE2 + SSSE3 instruction set. If you compile a function on one CPU and then execute it on another CPU, LLVM will probably crash
-
I’m not confident that the way I use to detect which instruction set your CPU uses is reliable. I’ve only tested it on a few laptops.