# Multiple GPUs with Julia

**URL:** <https://discourse.julialang.org/t/multiple-gpus-with-julia/11248>\
**Category:** GPU\
**Tags:** announcement\
**Created:** [May 29, 2018, 7:13pm UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248 "2018-05-29T19:13:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jacobcvt12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jacobcvt12/32/2695_2.png) [@jacobcvt12](https://discourse.julialang.org/u/jacobcvt12)\
**Post date:** [May 29, 2018, 7:13pm UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/1 "2018-05-29T19:13:25Z")

</div>

Has anyone every used multiple GPUs with Flux? If so, did you have to modify your code to run on multiple GPUs?

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [May 29, 2018, 7:58pm UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/2 "2018-05-29T19:58:43Z")

</div>

Yes, I use Flux with multiple GPUs. But my approach is not elegant.

In CUDAnative (commit f60c4754356225151c866da0ca512b434aa03abd) in src/CUDAnative.jl on line 75 I replaced `default_device[] = CuDevice(0)` with `dev = haskey(ENV, "CUDADEV") ? parse(ENV["CUDADEV"]) : 0; default_device[] = CuDevice(dev)`, after which I can (in bash) `export CUDADEV=1 && julia` to open a julia session with CUDAnative using GPU 1.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [May 29, 2018, 8:06pm UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/3 "2018-05-29T20:06:15Z")

</div>

You’ll be able to do `device!(::CuDevice)` on Julia 0.7 (technically, on any recent CUDAdrv/CUDAnative, but those are Julia 0.7 only). Do note that currently CuArray allocations are tied to a device, for better usability we should probably move to unified memory.

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [February 13, 2019, 1:56am UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/4 "2019-02-13T01:56:52Z")

</div>

Is there any live examples?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [February 13, 2019, 7:49am UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/5 "2019-02-13T07:49:00Z")

</div>

Only [https://github.com/JuliaGPU/CUDAnative.jl/blob/master/examples/multigpu.jl](https://github.com/JuliaGPU/CUDAnative.jl/blob/master/examples/multigpu.jl), but that uses unified memory which you probably don’t need. You could try and put all code that allocates and execute on the GPU within `device!` blocks.

---

<div class="post-metadata">

**Author:** ![maxfreu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxfreu/32/17468_2.png) [@maxfreu](https://discourse.julialang.org/u/maxfreu)\
**Post date:** [August 8, 2021, 7:44am UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/6 "2021-08-08T07:44:13Z")

</div>

Has there been some prigress wrt multi gpu trainig with Flux in the meantime? A mwe or tutorial would be nice, showing distributed training on multiple nodes with multiple gpus each.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [August 8, 2021, 10:57am UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/7 "2021-08-08T10:57:58Z")

</div>

[https://cuda.juliagpu.org/stable/usage/multigpu/](https://cuda.juliagpu.org/stable/usage/multigpu/)

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 8, 2021, 4:48pm UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/8 "2021-08-08T16:48:37Z")

</div>

For Flux in particular, you’ll probably want a combination of [https://github.com/DhairyaLGandhi/DaggerFlux.jl](https://github.com/DhairyaLGandhi/DaggerFlux.jl) for the user-facing API and [https://github.com/JuliaGPU/DaggerGPU.jl/pull/17](https://github.com/JuliaGPU/DaggerGPU.jl/pull/17) for the GPU transport side of things. If you have access to a CUDA-aware MPI, something like [https://github.com/AStupidBear/DistributedFlux.jl](https://github.com/AStupidBear/DistributedFlux.jl) is also an option.

---

<div class="post-metadata">

**Author:** ![maxfreu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxfreu/32/17468_2.png) [@maxfreu](https://discourse.julialang.org/u/maxfreu)\
**Post date:** [August 9, 2021, 9:49am UTC](https://discourse.julialang.org/t/multiple-gpus-with-julia/11248/9 "2021-08-09T09:49:25Z")

</div>

Thanks! These are really fresh! The fusion of Flux and Dagger was what I had hoped for, doing the low level work for me 🙂 I’ll try it out with GPUs when I find the time.