Multiple GPUs with Julia



Has anyone every used multiple GPUs with Flux? If so, did you have to modify your code to run on multiple GPUs?


Yes, I use Flux with multiple GPUs. But my approach is not elegant.

In CUDAnative (commit f60c4754356225151c866da0ca512b434aa03abd) in src/CUDAnative.jl on line 75 I replaced default_device[] = CuDevice(0) with dev = haskey(ENV, "CUDADEV") ? parse(ENV["CUDADEV"]) : 0; default_device[] = CuDevice(dev), after which I can (in bash) export CUDADEV=1 && julia to open a julia session with CUDAnative using GPU 1.


You’ll be able to do device!(::CuDevice) on Julia 0.7 (technically, on any recent CUDAdrv/CUDAnative, but those are Julia 0.7 only). Do note that currently CuArray allocations are tied to a device, for better usability we should probably move to unified memory.


Is there any live examples?


Only, but that uses unified memory which you probably don’t need. You could try and put all code that allocates and execute on the GPU within device! blocks.