Has anyone every used multiple GPUs with Flux? If so, did you have to modify your code to run on multiple GPUs?
Yes, I use Flux with multiple GPUs. But my approach is not elegant.
In CUDAnative (commit f60c4754356225151c866da0ca512b434aa03abd) in src/CUDAnative.jl on line 75 I replaced
default_device = CuDevice(0) with
dev = haskey(ENV, "CUDADEV") ? parse(ENV["CUDADEV"]) : 0; default_device = CuDevice(dev), after which I can (in bash)
export CUDADEV=1 && julia to open a julia session with CUDAnative using GPU 1.
You’ll be able to do
device!(::CuDevice) on Julia 0.7 (technically, on any recent CUDAdrv/CUDAnative, but those are Julia 0.7 only). Do note that currently CuArray allocations are tied to a device, for better usability we should probably move to unified memory.