CUDA aware MPI works on system but not for Julia

And when you run it without mpiexec?
Moreover, you should run it with mpiexecjl as described in the link I shared above.