# Building an MPI-parallelized library with BinaryBuilder.jl using MPI.jl

**URL:** <https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629>\
**Category:** General Usage\
**Tags:** mpi, binarybuilder\
**Created:** [August 27, 2020, 3:45pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629 "2020-08-27T15:45:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 27, 2020, 3:45pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/1 "2020-08-27T15:45:31Z")

</div>

We would like to use an external, MPI-parallelized library in our Julia project that itself is parallelized using `MPI.jl`. A key requirement for us, however, is to keep everything within the Julia package-sphere, thus we would need the external library to be built using `BinaryBuilder.jl` and shipped as a package.

Therefore, we are wondering: Is it possible to built such an MPI-aware library with BB by having it depend on `MPI.jl` (or directly on `MPICH_jll.jl`)? Better yet, has anyone tried this before (either successfully or not) and could help by pointing out what to do and what not? As far as I can tell, at least the releases of `MPICH_jll.jl` ship not just with the library files but also with the header files, so _theoretically_ it could be possible.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [August 27, 2020, 4:27pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/2 "2020-08-27T16:27:02Z")

</div>

There are some libraries already built with `MPICH_jll`, for example see [the General registry](https://github.com/JuliaRegistries/General/search?q=MPICH_jll&unscoped_q=MPICH_jll)

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 27, 2020, 4:53pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/3 "2020-08-27T16:53:32Z")

</div>

> [@giordano](#):
>
> There are some libraries already built with `MPICH_jll` , for example see [the General registry](https://github.com/JuliaRegistries/General/search?q=MPICH_jll&unscoped_q=MPICH_jll)

Great news, thanks! A quick scan seems to indicate that all of these libraries exclude Windows as a platform, even though there is a `MicrosoftMPI_jll` package available. Do you know if this is by choice or if there is a technical reason that it is not possible to support both Windows and Unix-like systems? Would it be possible to, e.g., make our library depend on `MPI.jl` such that the “correct” MPI backend (`MPICH_jll` on Unix, `MicrosoftMPI_jll` on Windows) is automatically available?

Also, would you happen to know if the feature of `MPI.jl` to switch to system-provided MPI installations also transiently applies to a library built with BB in such a way?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [August 27, 2020, 5:01pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/4 "2020-08-27T17:01:24Z")

</div>

> [@sloede](#):
>
> A quick scan seems to indicate that all of these libraries exclude Windows as a platform, even though there is a `MicrosoftMPI_jll` package available.

`PETSc_jll` is [built for 64-bit Windows](https://github.com/JuliaPackaging/Yggdrasil/blob/65397a129d0c240474ae8d9265b649f41b2574e4/P/PETSc/build_tarballs.jl#L76)

> [@sloede](#):
>
> Also, would you happen to know if the feature of `MPI.jl` to switch to system-provided MPI installations also transiently applies to a library built with BB in such a way?

I don’t think so

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 27, 2020, 8:31pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/5 "2020-08-27T20:31:12Z")

</div>

> [@giordano](#):
>
> `PETSc_jll` is [built for 64-bit Windows](https://github.com/JuliaPackaging/Yggdrasil/blob/65397a129d0c240474ae8d9265b649f41b2574e4/P/PETSc/build_tarballs.jl#L76)

Oops, apparently I didn’t read the build script for PETSc carefully enough (only Windows for i686 seems to be excluded) 🙈 Thanks for the fast & helpful replies!

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [August 27, 2020, 9:12pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/6 "2020-08-27T21:12:13Z")

</div>

> [@sloede](#):
>
> Also, would you happen to know if the feature of `MPI.jl` to switch to system-provided MPI installations also transiently applies to a library built with BB in such a way?

No: the problem is MPI doesn’t define a standard ABI, so different libraries use different constants and types, see [Exposing MPI interface · Issue #550 · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/issues/550).

What I think is more reasonable is to try to support providing your own libraries: there isn’t really a good mechanism to do this yet though.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 27, 2020, 9:26pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/7 "2020-08-27T21:26:49Z")

</div>

> [@simonbyrne](#):
>
> No: the problem is MPI doesn’t define a standard ABI, so different libraries use different constants and types, see [Exposing MPI interface · Issue #550 · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/issues/550).

Ah yes, I recall that discussion from a while back. Well, I think for us there seem to be two options then:

1. Accept the restriction that any BB-built library that depends on MPI\*\_jll for the build will only work with the jll-provided MPI binaries but not with system MPI libraries.
2. Provide a mechanism in a library wrapper package similar to `MPI.jl`’s to allow dynamic switching to a system library that was built against the system MPI installation.

The latter seems like a pain to make it work though… Or what is your take on this, given that you seem to be main author of the respective part of [MPI.jl](https://github.com/JuliaParallel/MPI.jl/blob/master/deps/build.jl)?

(By the way, the library we are looking at is [p4est](https://github.com/cburstedde/p4est), a parallel octree implementation for AMR with load balancing.)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [August 27, 2020, 9:35pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/8 "2020-08-27T21:35:51Z")

</div>

Note that Julia v1.6 should have a well defined “Preferences” system for packages: [https://github.com/JuliaLang/Pkg.jl/pull/1835](https://github.com/JuliaLang/Pkg.jl/pull/1835). This can help for having consistent preferences across multiple packages, but you’ll still need to implement the mechanism to switch to different libraries yourself

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [August 27, 2020, 10:07pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/9 "2020-08-27T22:07:40Z")

</div>

> [@sloede](#):
>
> The latter seems like a pain to make it work though… Or what is your take on this, given that you seem to be main author of the respective part of [MPI.jl](https://github.com/JuliaParallel/MPI.jl/blob/master/deps/build.jl)?

Indeed, which is why I haven’t done anything yet. The hooks in MPI.jl are already a mess; getting it working for downstream binaries is going to be even harder (since one would need to make sure that all the libraries are linked together correctly).

One idea is to integrate with [spack](https://spack.io/), which seems to have a good handle on specifying these sorts of dependencies. However it’s not clear how best to integrate this with Julia’s Pkg system (e.g. would you need a Julia Pkg depot per spack environment?).

Thoughts, ideas or contributions are very much welcome,

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [August 27, 2020, 10:26pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/10 "2020-08-27T22:26:14Z")

</div>

> [@simonbyrne](#):
>
> One idea is to integrate with [spack](https://spack.io/), which seems to have a good handle on specifying these sorts of dependencies.

I’ve thought a few times already about having an interface to spack (maybe to somewhat easily generate override files), but my understanding is that they only provide the command-line interface and there is no python package to call in, is that correct?

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [August 27, 2020, 10:37pm UTC](https://discourse.julialang.org/t/building-an-mpi-parallelized-library-with-binarybuilder-jl-using-mpi-jl/45629/11 "2020-08-27T22:37:56Z")

</div>

I’m not really sure: it does have an API ([spack package — Spack 0.19.0.dev0 documentation](https://spack.readthedocs.io/en/latest/spack.html)), but it doesn’t look like it is meant to really be used to run it.