Custom transport for Distributed.jl to utilize Infiniband and avoid MPI?

[fgerick]

Hi,
I was wondering if there’s any prospect of Distributed.jl and/or Dagger.jl to support interconnects such as Infiniband on HPC clusters?

There’s a nice summary of the downsides of MPI.jl and Distributed.jl in this post:

Distributed.jl vs MPI.jl

They are somewhat different models of parallelism: MPI.jl is based on a single-program multiple-data (SPMD) model, where (typically) all processes run the same program and periodically perform some communication operation to other processes. As a very well-established paradigm, lots of time and money have been spent optimizing these operations, both in terms of hardware (MPI is able to leverage fast interconnects, such as Infiniband), as well as software support (it ultimately underlies almos…

with the main downside of Distributed.jl being the missing Infiniband support. It would be great to circumvent going to MPI only for this.

I found only one post addressing this, saying that a custom transport could be implemented, but is not easily done:

Implementing parallel fluids code in Julia (parallel FFTW, MPI.jl, ...)

The Cluster Managers with custom transports section in the Julia documentation has a little information on creating custom transport layers for Julia’s native distributed memory parallelism. There’s an example in the Julia distribution of an implementation using the zmq library. I’ve had a thorough look through the internet for an infiniband implementation but haven’t been able to find one. I looked into doing it myself and it looks like a lot of work. However, I know very little about networki…

Now I’m no expert on any of this, but I think if the Julia internal Distributed.jl could leverage Infiniband interconnects it would be a big advantage. Especially to those people (like me) who do not want to rethink their whole code in MPI-style. I’d be happy to hear if there are any projects or intentions to support this!

Thanks

[ToucheSir]

Shooting in the dark, would https://github.com/JuliaParallel/UCX.jl work here?

[fgerick]

I don’t know UCX and what it does exactly, the ucx github page mentions Infiniband communication though. If I look at the UCX.jl example it does not look like a significant improvement over MPI.jl? Or do you mean to use ucx as an example for an implementation of the transport for Distributed.jl?

[jpsamaroo]

UCX implements support for a large number of network fabrics; OpenMPI uses UCX to access those fabrics and other communication schemes. I think that UCX.jl will probably be the future of custom transports for Distributed data transfer.

@vchuravy has been working on getting UCX.jl to underpin Distributed, and has had recent success with implementing a UCX-backed IOStream that Distributed should be able to use.

[charlescolley]

Do you have any pointers on where I can learn more about supporting Infiniband in pure julia?

I have a project which is currently suffering from the low bandwidth of TCP connections and I’d like to write something which can support larger bandwidth which could be used by Distributed.jl. I’m admittedly new to writing something like this (my only experience is working with UDP + TCP connections in a networking course) but I’d like to learn. I’ve had a look at the relevant ClusterManager documentation, the clustermanager/0mq example mentioned within, and the UCX.jl package, but I’m still unable to properly determine what fully needs to be written before I can support Infiniband.

Using UCX.jl to underpin the communication of Distributed.jl seems like the most promising first step (given the preliminary results in the last slide of this presentation), but the documentation in the package is sparse and I’m not sure how to integrate it.