I am happy and excited to share a recent publication where I showcase a software library called, LMDiskANN.jl (Low Memory Disk Approximate Nearest Neighbor Search). It is implemented in Julia Lang and allows users to store and search through large collections of vectors (embeddings) that can exceed the amount of RAM since it mostly resides on disk. The searches are done on a flexible graph connectivity set that allows for both insertions and deletions.
Here is the link to the paper,
https://joss.theoj.org/papers/10.21105/joss.08199
and the software repository,
Even better, today it was integrated into the general repository for Julia Lang so it can be added and used via ‘LMDiskANN’.
Key Features:
- Allows for both Insertions and Deletions
- Index resides on disk and allows for indexes that are larger than the capacity of RAM (memory) since the file is memory mapped
- Follows an implementation that is at the core based on LMDiskANN/FreshDiskANN making its insertions, searches etc efficient on a graph structure
- Users can associate their own key names for each vector inserted