Multiple GPUs with Julia

announcement

#1

Has anyone every used multiple GPUs with Flux? If so, did you have to modify your code to run on multiple GPUs?


#2

Yes, I use Flux with multiple GPUs. But my approach is not elegant.

In CUDAnative (commit f60c4754356225151c866da0ca512b434aa03abd) in src/CUDAnative.jl on line 75 I replaced default_device[] = CuDevice(0) with dev = haskey(ENV, "CUDADEV") ? parse(ENV["CUDADEV"]) : 0; default_device[] = CuDevice(dev), after which I can (in bash) export CUDADEV=1 && julia to open a julia session with CUDAnative using GPU 1.


#3

You’ll be able to do device!(::CuDevice) on Julia 0.7 (technically, on any recent CUDAdrv/CUDAnative, but those are Julia 0.7 only). Do note that currently CuArray allocations are tied to a device, for better usability we should probably move to unified memory.


#4

Is there any live examples?


#5

Only https://github.com/JuliaGPU/CUDAnative.jl/blob/master/examples/multigpu.jl, but that uses unified memory which you probably don’t need. You could try and put all code that allocates and execute on the GPU within device! blocks.