I’m excited to announce the v0.1.0 release of ReactantServer.jl, an inference server for models compiled with Reactant.jl!
From a single desktop GPU to a multi-node GPU cluster, ReactantServer has you covered:
- Load more models than fit on your GPU: model weights hot swap at PCIe speed, so you can serve far more models than fit in device memory. Gaming and workstation class GPUs can effectively serve as many models as fit in system RAM.
- KServe V2 Inference Protocol: call ReactantServer from any language with a gRPC client. We also provide ReactantServerClient.jl, tuned for low overhead and memory efficiency, with shared memory support for model inputs and outputs.
- Pre/post-processing in native Julia: already compiled your model but forgot to add a softmax output alongside your logits? Declare a second output in pure Julia right alongside your model.
- Scale up to multiple GPUs: ReactantServerGateway.jl sits in front of many ReactantServer workers as a load balancer. It can redistribute models across devices to balance compute and memory, or use standard strategies such as round-robin and least-outstanding (by requests or estimated compute time). Replica counts per model and scheduler behavior are configurable, including encouraging batch coalescing.
- Compile and export Lux.jl and PyTorch models with ReactantServerExport.jl; Support for Jax and other frameworks is something we are looking into for future releases, but in principle if you can express your model as a pure function of inputs and weights, you can easily export and serve it today if it compiles with Reactant.
- Simple unified configuration: the same node configuration syntax works whether you run ReactantServer.jl alone or use ReactantServerNode.jl to spin up many workers and a gateway automatically.
- Regulated profile for deterministic inference: modern inference stacks have many sources of nondeterminism. The regulated profile ensures that on the same hardware and software versions, every input maps to the same output.
Getting Started
You can use ReactantServer directly from Julia via Pkg in your own environment, or use the container image built for every release; tutorials for the container style deployment here: single gpu & multi gpu
Future Plans
- We do plan on supporting local LLM serving in the future, but the initial focus of the project was on smaller vision and language models.
- Make storing weights in RAM optional for workloads that can fit everything across the available GPUs
- PRs welcome!
Acknowledgements
- All of the Reactant/Enzyme/Enzyme-JAX contributors for the foundation this project sits on and for putting up with me as I continue to learn more about Reactant’s internals through trial and error.
- Medical Metrics Inc. for funding the projects development and allowing it to be open sourced.