UCX implements support for a large number of network fabrics; OpenMPI uses UCX to access those fabrics and other communication schemes. I think that UCX.jl will probably be the future of custom transports for Distributed data transfer.
@vchuravy has been working on getting UCX.jl to underpin Distributed, and has had recent success with implementing a UCX-backed IOStream that Distributed should be able to use.