# Vector Database in Julia: bad idea?

**URL:** <https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371>\
**Category:** Data\
**Tags:** question, idea, database\
**Created:** [April 12, 2023, 12:12am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371 "2023-04-12T00:12:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![gvdr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gvdr/32/6387_2.png) [@gvdr](https://discourse.julialang.org/u/gvdr)\
**Post date:** [April 12, 2023, 12:12am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/1 "2023-04-12T00:12:59Z")

</div>

Floating an idea:

has anybody tried to build a vector database (think Pinecone or Weaviate) in pure Julia?

If not, is there a specific reason why it’s a bad idea?

---

<div class="post-metadata">

**Author:** ![Eric\_Sadit\_Tellez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_sadit_tellez/32/33615_2.png) [@Eric\_Sadit\_Tellez](https://discourse.julialang.org/u/Eric_Sadit_Tellez)\
**Post date:** [April 12, 2023, 4:52am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/2 "2023-04-12T04:52:05Z")

</div>

I don’t know if there is any alternative to these vector databases, but several approximate nearest-neighbor algorithms exist around the ecosystem. For instance,

[https://github.com/sadit/SimilaritySearch.jl](https://github.com/sadit/SimilaritySearch.jl)

its `SearchGraph` structure is similar to HNSW, but with auto-tuning (easy/automatic hyperparameter setup), it supports incremental construction (insertions) and some similarity-based operations (single and batched nn search, allknn, etc.).

In particular, I think that Julia is a very interesting language for creating a similar search engine, e.g., different metric functions (`Distances.jl`) and fast user-defined distance functions.

It would be interesting to know the minimal requirements to create a vector database and see if achieving it with a reasonable effort is possible.

---

<div class="post-metadata">

**Author:** ![gvdr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gvdr/32/6387_2.png) [@gvdr](https://discourse.julialang.org/u/gvdr)\
**Post date:** [April 12, 2023, 10:08pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/3 "2023-04-12T22:08:38Z")

</div>

Totally agree with you. Both in terms of expansion potential (in Julia it becomes trivial to define ad hoc algos for the embeddings) and performance Julia would seem a natural choice.

Instead, I see most of the vector DBs going for Go (pun not intended).

Maybe I’ll give it a go. A use case less explored, to start with, would be storing nodes of a graph as embedded points in a latent space.

But I’m a mathematician with zero experience in building databases, so every help and suggestion is welcome 🤗

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [April 12, 2023, 11:55pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/4 "2023-04-12T23:55:30Z")

</div>

I have no idea what’s behind a Vector Database. However, my experience can help 🙂

Some time ago, we did not have a pure Julia SGP4 orbit propagation. We had one package that interfaces with a Python library ([GitHub - crbinz/SGP4.jl: Julia wrapper of the Python SGP4 library](https://github.com/crbinz/SGP4.jl)). It was working pretty well! However, after I reimplemented the SGP4 in Julia, the speed gain was enormous (at least using the Julia version back in the day). It reached the speed of the SGP4 C library, which is just amazing.

Hence, we may not have a pure Julia implementation of a Vector Database because nobody needed to go through the endeavor. However, if you want to try, it can really pay off big time 🙂

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [April 13, 2023, 11:24am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/5 "2023-04-13T11:24:42Z")

</div>

> [@Eric\_Sadit\_Tellez](#):
>
> its `SearchGraph` structure is similar to HNSW, but with auto-tuning

FYI: There is HNSW.jl available, and HNSW is I believe the key feature of Pinecone. I’m rather new to this, so are you saying the other package is better?

> [@gvdr](#):
>
> why it’s a bad idea?

It’s certainly possible and I’m not sure it’s a “bad” idea, just not needed; is there a real need to reinvent the wheel in Julia? There is already Pinecone.jl to access that DB. I’m assuming that database doesn’t run in-process. That’s usually a good thing. For in-process databases, such as SQLite (and DuckDB), written in C, you are taking a security risk, or really trusting the DB is bug-free, or at least has no memory-corrupting bugs. If you do this in Julia (and it’s in-process), then because of bounds-checking be default, it’s less of a risk, then the DB must not disable them (or if really not screw up), but it goes goth ways, all your other code has also that same constraint. [Well you can enable bounds-checking globally, then you would be safe, unless you include e.g. some C code, JLLs.]

While Googling I got this result:

> **[Community | Pinecone](https://www.pinecone.io/community/)**
>
> Search through billions of items for similar matches to any object, in milliseconds. It’s the next generation of search, an API call away.

> Web **Pinecone**.jl is a **Julia** API for the **Pinecone** vector database. ML Q&A The Q&A tool takes discussions and docs from some of the best Python ML libraries and collates their …

Clicking on the link I no longer see the Julia package mentioned on that page, but at least it was (and maybe still is elsewhere), so I assume it’s good. I see the package author mentioned it here, and he seems very close with the company:

> **[Use Pinecone to See Which Parent Most Resembles Your Kids - Menlo Ventures](https://menlovc.com/perspective/use-pinecone-to-see-which-parent-most-resembles-your-kids)**
>
> Menlo Ventures believes the next decade of enterprise apps will use semantic search, whether it’s security, observability, or e-commerce.

> This is an example of trying out VGG from within DeepFace, which has a vector length of 2,622

I’ve seen elsewhere that GPT-3 has over 1500 dimensions, not sure about in latest ChatGPT/GPT-4, but at least people go higher than that. Is there some inherent limit in the database, or a reason to go much higher? I’m just curious what’s the highers number of dimensions people use, and the pros and cons of many dims.

---

<div class="post-metadata">

**Author:** ![Eric\_Sadit\_Tellez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_sadit_tellez/32/33615_2.png) [@Eric\_Sadit\_Tellez](https://discourse.julialang.org/u/Eric_Sadit_Tellez)\
**Post date:** [April 13, 2023, 2:56pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/6 "2023-04-13T14:56:10Z")

</div>

The algorithm behind `SimilarySearch.jl` (`SearchGraph`) is competitive and, in some cases, improves than FAISS’s HNSW. Note that we also compared it with Google’s SCANN, which is very fast.

> **[\`SimilaritySearch.jl\`: Autotuned nearest neighbor indexes for Julia](https://joss.theoj.org/papers/10.21105/joss.04442)**
>
> Tellez et al., (2022). \`SimilaritySearch.jl\`: Autotuned nearest neighbor indexes for Julia. Journal of Open Source Software, 7(75), 4442, https://doi.org/10.21105/joss.04442

> **[2201.07917.pdf](https://arxiv.org/pdf/2201.07917.pdf)**
>
> 1047.47 KB

In any case, the auto-tuning gives an advantage for inexperienced users, i.e., the user only declares the objective recall, and the algorithm will try to achieve and maintain it along the life of the index. Note that always declaring maximum recall is not a good deal since it trades between speed and accuracy. The same characteristic reduces the indexing time since the index is only made once.

Perhaps a new vector database is too much work, but maybe not if we isolate functionality and reuse the ecosystem. Perhaps most people only need to store datasets in hdf5 and then use a regular similarity search algorithm. I never used Pinecone or Weaviate, but I perform benchmarks frequently with FAISS, SCANN, and others, so I don’t have the complete picture here.

---

<div class="post-metadata">

**Author:** ![Eric\_Sadit\_Tellez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_sadit_tellez/32/33615_2.png) [@Eric\_Sadit\_Tellez](https://discourse.julialang.org/u/Eric_Sadit_Tellez)\
**Post date:** [April 13, 2023, 5:00pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/7 "2023-04-13T17:00:14Z")

</div>

On the other hand, maybe people want to use/consume it from other languages. Therefore, a tutorial on preparing data, storing vectors in hdf5 (or something similar), indexing, and exposing search functions with a REST/JSON API could be enough.

It could be interesting to have an all-in-one solution, as others pointed out, it can be much more complicated and perhaps require a whole-team full-time effort, and that requires a clearer motivation.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [April 13, 2023, 5:56pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/8 "2023-04-13T17:56:25Z")

</div>

Thanks for me mentioning this library. Have you tried composing it with e.g. [GitHub - baggepinnen/DynamicAxisWarping.jl: Dynamic Time Warping (DTW) and related algorithms in Julia, at Julia speeds](https://github.com/baggepinnen/DynamicAxisWarping.jl) in order to search and cluster time series dara?

---

<div class="post-metadata">

**Author:** ![Eric\_Sadit\_Tellez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_sadit_tellez/32/33615_2.png) [@Eric\_Sadit\_Tellez](https://discourse.julialang.org/u/Eric_Sadit_Tellez)\
**Post date:** [April 14, 2023, 1:52pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/9 "2023-04-14T13:52:31Z")

</div>

No, I haven’t used DTW (nor the package). It would be interesting to see if we can take advantage of composing. I understand the DTW is not a proper metric, and some tricks are needed. Perhaps, using the `ExhaustiveSearch` struct (brute force search instead of indexing) with multithreading can be enough to see if we can obtain some interesting results.

Of course, I was not exhaustive, but I found the following paper that shows the benefit of using metric indexes for the task.

> **[0811.3301.pdf](https://arxiv.org/pdf/0811.3301.pdf)**
>
> 447.62 KB

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [April 15, 2023, 6:59am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/10 "2023-04-15T06:59:34Z")

</div>

Thanks for the thoughts and the paper. That‘s something I overlooked, it changes the picture.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 29, 2023, 9:53am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/11 "2023-10-29T09:53:45Z")

</div>

Is it a prerequisite to have a good embeddings models runnable in Julia? I guess technically it can just take some vector computed by other models, but practically I can’t imagine Julia beating C++/Rust in this type of narrow, user-agnostic tasks (a few numbers in a few numbers out). So if we have a good stack of other things to go around the vector DB it would may more sense to choose Julia to begin with

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [October 29, 2023, 1:37pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/12 "2023-10-29T13:37:39Z")

</div>

I’m not sure if this would suit your purposes but bringing [LanceDB](https://lancedb.com/) into Julia might be faster and it integrates with other open source tools through arrow. It’s written in rust and is an embedded database rather than a service, but only has python and JavaScript clients now.

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 6:06pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/13 "2023-10-31T18:06:37Z")

</div>

Is LanceDB serverless like SQLite?

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [October 31, 2023, 6:21pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/14 "2023-10-31T18:21:54Z")

</div>

Yes. The format is explained [here](https://github.com/lancedb/lance).

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 6:47pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/15 "2023-10-31T18:47:36Z")

</div>

> [@rdavis120](#):
>
> here

It looks interesting. For a lot of things it might not have an advantage over relational DBs, but I can see where it could be useful for machine learning. I’m wondering if Julia should have the ablility to call Rust (and possibly Zig) libraries, the same way it can call C or Fortran libraries.

---

<div class="post-metadata">

**Author:** ![gvdr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gvdr/32/6387_2.png) [@gvdr](https://discourse.julialang.org/u/gvdr)\
**Post date:** [October 31, 2023, 10:35pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/16 "2023-10-31T22:35:36Z")

</div>

Well, calling Rust should be possible just with ccall(), right?

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [November 9, 2023, 9:31pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/17 "2023-11-09T21:31:32Z")

</div>

OK, I’ll try it and see.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [December 1, 2023, 7:43am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/18 "2023-12-01T07:43:16Z")

</div>

I’m pretty interested in this idea. `SimilaritySearch.jl` seems like a pretty good component of a vector DB, what I’m struggling to find a nice solution to is getting text embeddings and running queries with it.

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [May 20, 2024, 7:56pm UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/19 "2024-05-20T19:56:04Z")

</div>

Just saw this one. I’m slowly trying to build a graph + vector DB from the ground up.

> [@tecosaur](#):
>
> I’m pretty interested in this idea. `SimilaritySearch.jl` seems like a pretty good component of a vector DB, what I’m struggling to find a nice solution to is getting text embeddings and running queries with it.

This is relatively well solved with PromptingTools.jl, I’ve been able to use Ollama to generate my own embeddings, but it supports OpenAI, nomic, etc.

I’m mostly messing around with the storage layer right now, i.e. how to serialize relationships, nodes, and edges. I have a very simple vector search built in but SimilaritySearch.jl looks _awesome_ and I hadn’t noticed it before!

If anyone’s interested I’m considering scaling it up as a potential commercial enterprise and/or simple learning project. Happy to take people on, I suspect it’d be a good open source platform.

The goals are

- Implement the GQL standard (similar to cypher)
- Support a first-class Julia interface, i.e. you should be able to use it with no query language at all
- High horizontal scalability
- Ultra-simple, no-nonsense, distributed vector search
- Do it from scratch because I like learning things that way

I do think that Julia is a very good candidate for this kind of thing – making the internals of the database operate on type of the type system seems super flexible, especially for heterogeneously typed nodes/edges/etc.

---

<div class="post-metadata">

**Author:** ![gvdr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gvdr/32/6387_2.png) [@gvdr](https://discourse.julialang.org/u/gvdr)\
**Post date:** [May 21, 2024, 5:00am UTC](https://discourse.julialang.org/t/vector-database-in-julia-bad-idea/97371/20 "2024-05-21T05:00:40Z")

</div>

Count me in