NVLink's Compatibility with KernalAbstractions

Specific Domains GPU
1

I am planning to build a workstation with two NVIDIA GPUs (A6000-ish) connected with NVLink since we would like to run some memory-intensive tasks – high-resolution CFD simulation (WaterLily.jl parallelized by KernalAbstractions.jl)

I would like to know if anyone has successfully run some KernalAbstraction codes upon several GPUs with NVLink. (According to NVIDIA advertisement, program seems able to access every GPU’s cuda cores and memory simultaneously as if they are one GPU.)

If so, is it also possible to disable NVLink so that we could also test the compatiblity of multi-GPU version?

2

Multi-GPU works fine with KA.jl + MPI.jl (no “native” support for multi-GPU in KA though). I don’t know for the NVlink thing. If with NVlink the GPUs are exposed as a single device, I guess it may work, but there is no specific support for this in KA neither. I guess this is something to handle outside Julia.

1 Like
3

FWIW, CUDA.jl automatically uses NVLink or whatever’s available when copying between arrays on different devices: CUDA.jl 3.5-3.8 ⋅ JuliaGPU, so you don’t need MPI.jl per se.

4

Ah Thank you for the reply. Then I guess KA is compatible with NVLINK since CUDA allows it.