# Multi-GPU inference in Flux.jl

**URL:** https://discourse.julialang.org/t/multi-gpu-inference-in-flux-jl/135210
**Category:** General Usage
**Tags:** gpu, cuda, flux
**Created:** [January 22, 2026, 10:05pm UTC](https://discourse.julialang.org/t/multi-gpu-inference-in-flux-jl/135210 "2026-01-22T22:05:17Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Chrysoberyl](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@Chrysoberyl](https://discourse.julialang.org/u/Chrysoberyl)
#### Post date: [January 22, 2026, 10:05pm UTC](https://discourse.julialang.org/t/multi-gpu-inference-in-flux-jl/135210/1 "2026-01-22T22:05:17Z")

</div>

Flux allows for parallel training with multiple GPUs: [GPU Support · Flux](https://fluxml.ai/Flux.jl/dev/guide/gpu/#Distributed-data-parallel-training)

In my use case, I need to run inference of a single model on multiple GPUs. Is there a way to load balance inference calls to the model so all GPUs are maximally utilized?

---

<div class="post-metadata">

### Author: ![Noob](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noob/32/6595_2.png) [@Noob](https://discourse.julialang.org/u/Noob)
#### Post date: [January 23, 2026, 11:06pm UTC](https://discourse.julialang.org/t/multi-gpu-inference-in-flux-jl/135210/2 "2026-01-23T23:06:04Z")

</div>

I could be wrong, but I expect you would have to roll your own here. Out of curiosity, what sort of model and what sort of inference is it?

---

<div class="post-metadata">

### Author: ![Chrysoberyl](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@Chrysoberyl](https://discourse.julialang.org/u/Chrysoberyl)
#### Post date: [January 24, 2026, 12:07am UTC](https://discourse.julialang.org/t/multi-gpu-inference-in-flux-jl/135210/3 "2026-01-24T00:07:26Z")

</div>

I’m just running a small GNN on a graph classification task