We would like to use an external, MPI-parallelized library in our Julia project that itself is parallelized using MPI.jl. A key requirement for us, however, is to keep everything within the Julia package-sphere, thus we would need the external library to be built using BinaryBuilder.jl and shipped as a package.
Therefore, we are wondering: Is it possible to built such an MPI-aware library with BB by having it depend on MPI.jl (or directly on MPICH_jll.jl)? Better yet, has anyone tried this before (either successfully or not) and could help by pointing out what to do and what not? As far as I can tell, at least the releases of MPICH_jll.jl ship not just with the library files but also with the header files, so theoretically it could be possible.
Great news, thanks! A quick scan seems to indicate that all of these libraries exclude Windows as a platform, even though there is a MicrosoftMPI_jll package available. Do you know if this is by choice or if there is a technical reason that it is not possible to support both Windows and Unix-like systems? Would it be possible to, e.g., make our library depend on MPI.jl such that the “correct” MPI backend (MPICH_jll on Unix, MicrosoftMPI_jll on Windows) is automatically available?
Also, would you happen to know if the feature of MPI.jl to switch to system-provided MPI installations also transiently applies to a library built with BB in such a way?
Oops, apparently I didn’t read the build script for PETSc carefully enough (only Windows for i686 seems to be excluded) Thanks for the fast & helpful replies!
Ah yes, I recall that discussion from a while back. Well, I think for us there seem to be two options then:
Accept the restriction that any BB-built library that depends on MPI*_jll for the build will only work with the jll-provided MPI binaries but not with system MPI libraries.
Provide a mechanism in a library wrapper package similar to MPI.jl’s to allow dynamic switching to a system library that was built against the system MPI installation.
The latter seems like a pain to make it work though… Or what is your take on this, given that you seem to be main author of the respective part of MPI.jl?
(By the way, the library we are looking at is p4est, a parallel octree implementation for AMR with load balancing.)
Note that Julia v1.6 should have a well defined “Preferences” system for packages: https://github.com/JuliaLang/Pkg.jl/pull/1835. This can help for having consistent preferences across multiple packages, but you’ll still need to implement the mechanism to switch to different libraries yourself
Indeed, which is why I haven’t done anything yet. The hooks in MPI.jl are already a mess; getting it working for downstream binaries is going to be even harder (since one would need to make sure that all the libraries are linked together correctly).
One idea is to integrate with spack, which seems to have a good handle on specifying these sorts of dependencies. However it’s not clear how best to integrate this with Julia’s Pkg system (e.g. would you need a Julia Pkg depot per spack environment?).
Thoughts, ideas or contributions are very much welcome,
I’ve thought a few times already about having an interface to spack (maybe to somewhat easily generate override files), but my understanding is that they only provide the command-line interface and there is no python package to call in, is that correct?
Thanks for the fast & helpful replies!