# CUDA aware MPI works on system but not for Julia

**URL:** <https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060>\
**Category:** Julia at Scale\
**Tags:** parallel, mpi\
**Created:** [January 23, 2022, 9:39am UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060 "2022-01-23T09:39:06Z")\
**Posts on this page:** 11\
**Page:** 2

<div class="post-metadata">

**Author:** ![Chiil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiil/32/27474_2.png) [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Post date:** [January 23, 2022, 9:50pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/21 "2022-01-23T21:50:09Z")

</div>

Disabling the memory pool did the trick, thanks!. All other suggestions I already tested thanks to the great help of @samo. I do not know which conclusion to draw from this, though. Does that mean that the memory pool and CUDA-aware MPI are incompatible together? Or is this something that needs fixing in our MPI installation, or in `MPI.jl` or `CUDA.jl`?

---

<div class="post-metadata">

**Author:** ![luraess](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luraess/32/16189_2.png) [@luraess](https://discourse.julialang.org/u/luraess)\
**Post date:** [January 23, 2022, 10:41pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/22 "2022-01-23T22:41:41Z")

</div>

Glad this did the trick!

> Does that mean that the memory pool and CUDA-aware MPI are incompatible together?

Yes, it seems that with CUDA-aware MPI, memory management by CUDA.jl “conflicts” with what CUDA-aware MPI expects (see [https://juliagpu.gitlab.io/CUDA.jl/usage/memory/#Memory-pool](https://juliagpu.gitlab.io/CUDA.jl/usage/memory/#Memory-pool) and [https://juliagpu.gitlab.io/CUDA.jl/usage/memory/#Environment-variables](https://juliagpu.gitlab.io/CUDA.jl/usage/memory/#Environment-variables)). Setting the memory pool to `none` “directly defers to the CUDA allocator”.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 24, 2022, 7:22am UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/23 "2022-01-24T07:22:48Z")

</div>

> [@luraess](#):
>
> memory management by CUDA.jl “conflicts” with what CUDA-aware MPI expects

Nothing CUDA.jl does here is fancy; rather CUDA’s own APIs are incompatible with itself: [Legacy cuIpc\* APIs incompatible with stream-ordered allocator · Issue #1053 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/issues/1053).

> [@Chiil](#):
>
> I wonder where this 11.6 CUDA driver comes from, `nvidia-smi` gives me: `NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 ` , so I do not know where this 11.6 originates from.

CUDA.jl upgrades your driver library by using a forward-compatible `libcuda.so`.

> [@samo](#):
>
> ```julia
> JULIA_CUDA_USE_BINARY_BUILDER=false
> 
> ```
> 
> needs to be set for each process.

Why? What’s inherently incompatible between our CUDA artifacts and MPI? The issue here seems the IPC/memory pool incompatibility, not the actual CUDA binaries.

---

<div class="post-metadata">

**Author:** ![Chiil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiil/32/27474_2.png) [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Post date:** [January 24, 2022, 8:03am UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/24 "2022-01-24T08:03:11Z")

</div>

Alright, that is very useful information. I can work without the memory pool for now, and hopefully this gets resolved in the future. If it helps to provide failing test programs’s anywhere let me know, but I conclude from your answer that the problem is already very clear to the developers.

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [January 24, 2022, 12:48pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/25 "2022-01-24T12:48:38Z")

</div>

> [@maleadt](#):
>
> What’s inherently incompatible between our CUDA artifacts and MPI?

I think there is no reason to believe that they would be inherently incompatible. It certainly is only a matter of the installation.

However, as far as I am aware, it is currently not possible to have CUDA-aware MPI.jl working without using a system installed CUDA-aware MPI, and in a CUDA-aware installation, we need to specify which CUDA installation to use. So, it seems natural to use also this very same system installed CUDA for CUDA.jl in order to be sure that all works smoothly. Now, it I could imagine that it is possible to either 1) first install CUDA.jl with artifacts and install CUDA-aware system MPI with it or 2) to just use the CUDA libraries from the CUDA.jl artifacts at runtime. I think 1) could be of interest in order to avoid having to install CUDA manually; I am not sure how much benefits or problems could bring 2). Do you have any coments on 1) and 2)?

In any case, I think most interesting would be, if at some point, MPI.jl could support CUDA-aware MPI without having to rely on a system-installed MPI. This would be nice for small clusters or multi-GPU workstations - at least for a quick-start or a fallback (on supercomputers, a system-optimized MPI - as Cray-MPICH in our case - will certainly always be preferred). @simonbyrne, could you maybe comment on the feasibility of this and if this is anywhere on the sky? 🙂

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 24, 2022, 1:18pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/26 "2022-01-24T13:18:44Z")

</div>

Ah ok, so the MPI back-end JLLs selected by MPI.jl (assuming JLLs are used, and it’s not just the system version again) need to build against the same CUDA version used by CUDA.jl. That’s a work in progress, and not possible yet (it needs a CUDA\_jll.jl that can be used by both CUDA.jl and those MPI back-ends).

> [@samo](#):
>
> I think 1) could be of interest in order to avoid having to install CUDA manually; I am not sure how much benefits or problems could bring

The other advantage is that in general CUDA.jl does a better job selecting a CUDA toolkit that’s supported by your system, both in terms of compatibility and in selecting the most up-to-date version, which matters since there’s known compilation bugs with all but the most recent CUDA compiler.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [January 24, 2022, 6:46pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/27 "2022-01-24T18:46:38Z")

</div>

> [@samo](#):
>
> In any case, I think most interesting would be, if at some point, MPI.jl could support CUDA-aware MPI without having to rely on a system-installed MPI. This would be nice for small clusters or multi-GPU workstations - at least for a quick-start or a fallback (on supercomputers, a system-optimized MPI - as Cray-MPICH in our case - will certainly always be preferred). @simonbyrne, could you maybe comment on the feasibility of this and if this is anywhere on the sky? 🙂

Possibly? @vchuravy did get UCX\_jll to build on a small number of platforms with CUDA support, but it’s not clear if this is feasible more generally.

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [January 24, 2022, 7:43pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/28 "2022-01-24T19:43:02Z")

</div>

Thanks @maleadt and @simonbyrne. It is good to have from time to time a review of the situation 🙂

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 24, 2022, 9:28pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/29 "2022-01-24T21:28:47Z")

</div>

Does this work have any connection to [https://github.com/JuliaPackaging/Yggdrasil/issues/2063](https://github.com/JuliaPackaging/Yggdrasil/issues/2063)? Asking because AFAIK MPI.jl provides the only GPU-compatible broadcast + allreduce interface in Julia land, but deep learning framework users are unlikely to have a compatible system MPI installation.

---

<div class="post-metadata">

**Author:** ![Chiil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiil/32/27474_2.png) [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Post date:** [January 24, 2022, 9:42pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/30 "2022-01-24T21:42:28Z")

</div>

No it hasn’t. I made a Julia port [https://github.com/Chiil/MicroHH.jl](https://github.com/Chiil/MicroHH.jl) of our C++/CUDA atmospheric simulator [https://github.com/microhh/microhh](https://github.com/microhh/microhh) and was trying how hard it is to get CUDA-aware MPI running.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [January 24, 2022, 10:15pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060/31 "2022-01-24T22:15:10Z")

</div>

Yes we should be able to handle that generally, once some of the artifact work has propagated a bit.

[Previous page](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060.md?page=1)