How to add/query a dict that contains tens of millions of items?

General Usage
1

I’m using a dict with struct key and struct value. Size of each key value pair is 0.5~3k bytes. The problem is that it can be accumulated to tens of millions of key-value pairs, which would cause OOM on my computer.
I use it as a cache, i.e. just add and query. I know disk based key-value store can do it. But the LevelDB and RocksDB libs of Julia were quite outdated. They don’t even pass Julia compiling.
Is there any other way I can try?

Thanks

2

If it’s supposed to be a cache, then maybe you don’t need to keep all items the whole time? If that’s the case, GitHub - JuliaCollections/LRUCache.jl: An implementation of an LRU Cache in Julia might help.

3

Here is a discussion of SQLite’s lsm1 extension. It perhaps can handle your use case. To manage everything with Julia, the SQLite.jl package is up-to-date, well maintained and quite efficient.

A similar approach using Rust (instead of Julia) is presented here.

4

Unfortunately, I have to keep all items for later usage. Those items will be saved to disk and loaded for next runnings.

5

Thanks. I’ll try it.