It turns out that BERT based models are trivial to export and serve with ReactantServer. I added tutorials here for four common use cases: Transformer Text Models · ReactantServer.jl
- Dense Embedding: sentence-transformers/all-MiniLM-L6-v2
- Sparse Embedding: prithivida/Splade_PP_en_v2
- Cross Encoders: cross-encoder/ms-marco-MiniLM-L6-v2
- Classification: distilbert/distilbert-base-uncased-finetuned-sst-2-english
If your lab/company uses a vector based knowledgebase / retrieval system for RAG, you can already host the models needed for that with ReactantServer.
I also looked into serving LLMs. The main challenges are potentially needing many different compiled program sizes with padding / KV cache using StableHLO semantics / wrapping all of that up together in a way that can be effectively batched. In principle, there shouldn’t be any reason why you couldn’t serve a dense model like Qwen3.6-27B today. I don’t have time to look into this currently, but serving these sorts of models which can reasonably fit on a single workstation class GPU is something I would like to support in the future. Contributions are welcome!