Multiple GPUs with Julia



Has anyone every used multiple GPUs with Flux? If so, did you have to modify your code to run on multiple GPUs?


Yes, I use Flux with multiple GPUs. But my approach is not elegant.

In CUDAnative (commit f60c4754356225151c866da0ca512b434aa03abd) in src/CUDAnative.jl on line 75 I replaced default_device[] = CuDevice(0) with dev = haskey(ENV, "CUDADEV") ? parse(ENV["CUDADEV"]) : 0; default_device[] = CuDevice(dev), after which I can (in bash) export CUDADEV=1 && julia to open a julia session with CUDAnative using GPU 1.


You’ll be able to do device!(::CuDevice) on Julia 0.7 (technically, on any recent CUDAdrv/CUDAnative, but those are Julia 0.7 only). Do note that currently CuArray allocations are tied to a device, for better usability we should probably move to unified memory.