How to assign a device for each worker correctly in a multi gpu multi node scenario?

Oh thank you for the clarification. The reason I am persuing an option that doesn’t rely on slurm is because I want to be able to write a code that uses both the GPUs and CPUs for compute. That way I would assign a gpu to each of the first workers in the node, and a few more workers that would some cpu stuff using Threads.

I will test the second approach you mentioned.

Thank you!