I have had the same issue on an LSF cluster. I think it should only happen when the same port is used with the same ip, something which can happen when a single machine has many slots.
Here is a post of how I worked around the above issue on LSF: How to configure which port workers listen - #6 by DrChainsaw
Dont know if slurm does things differently so that other constraints come to play.