# Distributed.jl, DistributedArray.jl with InfiniBand cluster
**URL:** https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049
**Category:** Julia at Scale
**Tags:** cluster, distributed
**Created:** [April 26, 2022, 11:56am UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049 "2022-04-26T11:56:29Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [April 26, 2022, 11:56am UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/1 "2022-04-26T11:56:29Z")
I’m developing an algorithm using Distributed.jl and DistributedArrays.jl, and I’m supposed to run this algorithm on a cluster supercomputer. I’ve recently been told that it uses Infiniband to connect internal nodes with each other. It’s the first time I do parallel computing, and I’ve never heard about Infiniband before, so I wanted to ask if I can keep using DistributedArrays.jl or if I have to switch to MPI.jl (which I would avoid, if possible). Sorry if this question is a bit nooby and vague, but I’m a bit lost. Searching online gave different answers and in general it’s not something that is a lot talked about online. And as I’ve said, it’s my first time dealing with these kind of things.
---
### Author:  [@pablosanjose](https://discourse.julialang.org/u/pablosanjose)
#### Post date: [April 26, 2022, 12:22pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/2 "2022-04-26T12:22:35Z")
I’m in a similar situation: I’m starting to look at deploying our simulations in clusters with several nodes connected by infiniband, so I look forward to any response from people with this kind of experience.
(As far as I understand, Infiniband is much like good old ethernet, just much faster, so you can reduce the bottleneck of internode communication.)
Possibly relevant: [Running Julia in a SLURM Cluster - #2 by CameronBieganek](https://discourse.julialang.org/t/running-julia-in-a-slurm-cluster/67614/2)
---
### Author:  [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)
#### Post date: [April 26, 2022, 1:41pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/3 "2022-04-26T13:41:45Z")
It looks like you’ll be a bit of a pioneer: [Custom transport for Distributed.jl to utilize Infiniband and avoid MPI?](https://discourse.julialang.org/t/custom-transport-for-distributed-jl-to-utilize-infiniband-and-avoid-mpi/74097)
UCX.jl seems to be the leading choice here, but it hasn’t gotten a new release in over a year, although several PRs appear to be pending. cc @vchuravy
If this is your first time doing parallel computing, it’s probably easiest to bite the bullet and use MPI rather than acting as a test dummy for a new protocol.
---
### Author:  [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)
#### Post date: [April 26, 2022, 1:46pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/4 "2022-04-26T13:46:21Z")
Perhaps you could also consider [PartitionedArrays.jl](https://github.com/fverdugo/PartitionedArrays.jl) which uses MPI as a backend.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [April 26, 2022, 1:53pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/5 "2022-04-26T13:53:39Z")
That’s a pity. Looking online I did found people using Julia with IPoIB, which why I asked if it was possible. Than again, knowing nothing (yet) about the clsuter I’m about to work on, I’m not sure if IPoIB even applies.
---
### Author:  [@vchuravy](https://discourse.julialang.org/u/vchuravy)
#### Post date: [April 26, 2022, 2:02pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/6 "2022-04-26T14:02:06Z")
You can use [MPIClusterManager.jl](https://github.com/JuliaParallel/MPIClusterManagers.jl) to run Distributed.jl over MPI.
My long-term plan is to build something akin to Distributed.jl using UCX as the communication layer.
But there is no ETA on that 🙂
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [April 26, 2022, 2:25pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/7 "2022-04-26T14:25:30Z")
Why do you need to use MPI? I’d say try with `Distributed.jl` and `DistributedArrays.jl` and use a cluster manager (say, `ClusterManagers.jl`). I am assuming your cluster has some workload management software already installed (e.g., Slurm, PBS).
My cluster is also connected with Infiniband, but to me that is just some backend magic to connect the nodes together. Instead, my workflow is simply to log in to my head node, start Julia, and use ClusterManagers to connect to the remaining nodes.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [April 26, 2022, 2:32pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/8 "2022-04-26T14:32:19Z")
That is what I thought too, but I wasn’t sure if using InfiniBand would have been a problem. I’ve never used a ClusterManager tho, I wouldn’t even know which one to choose.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [April 26, 2022, 2:35pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/9 "2022-04-26T14:35:26Z")
Do you know what you have installed on your cluster? Your path to parallelization could be as simple as
```julia
using ClusterManagers
addprocs(SlurmManager(n))
pmap(1:100) do
#code here
end
```
which will start distribute the code in the `do` block over `n` nodes. The `pmap` function will manage scheduling, allocation, and return your results in an array for you.
Note I am talking about embarrassingly parallel problems. Multithreading is a whole different beast.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [April 26, 2022, 2:49pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/10 "2022-04-26T14:49:35Z")
I still have to have access to the cluster. I’ll be sure to be back here once I know more. It’s good to know that I don’t necessarily need to use MPI.jl. I wanted to know if I had to rework my algorithm from scratch using MPI or not. Thank you.
---
### Author:  [@pablosanjose](https://discourse.julialang.org/u/pablosanjose)
#### Post date: [April 26, 2022, 5:09pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/11 "2022-04-26T17:09:04Z")
This pointer is really useful @affans, thanks! But I cannot give you more than one “heart” 😃
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [April 26, 2022, 5:46pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/12 "2022-04-26T17:46:34Z")
Distributed.jl and DistributedArrays.jl are not the only possibility if you do not want to programm using MPI.jl. There are high-level packages that build on MPI.jl which render distributed parallelization easier, e.g. ImplicitGlobalGrid.jl:
> **[GitHub - eth-cscs/ImplicitGlobalGrid.jl: Almost trivial distributed...](https://github.com/eth-cscs/ImplicitGlobalGrid.jl)**
>
> Almost trivial distributed parallelization of stencil-based GPU and CPU applications on a regular staggered grid - GitHub - eth-cscs/ImplicitGlobalGrid.jl: Almost trivial distributed parallelizatio...
High-level packages have of course often a certain focus and not each of them might be suited for your use case, which you have not mentioned though.
A Julia application that uses MPI.jl - whether directly or via a high-level package - can be submitted on a cluster pretty much the same way as any other MPI application, that might be written in C/C++/Fortran/… See here for an example on how you can run Julia applications, e.g., on the Piz Daint supercomputer at the Swiss National Supercomputing Centre:
[https://user.cscs.ch/tools/interactive/julia/#how-to-run-on-piz-daint](https://user.cscs.ch/tools/interactive/julia/#how-to-run-on-piz-daint)
---
### Author:  [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)
#### Post date: [April 26, 2022, 6:03pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/13 "2022-04-26T18:03:52Z")
In the case of embarrassingly parallel problems isn’t it better to have SLURM (or whatever) launch several julia processes?
In cluster, small jobs usually have higher priority, so it is usually better to launch many small jobs (each one starting julia an then working on a piece of the 1:100) than having one big job (which starts julia an then works on the whole 1:100).
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [April 26, 2022, 6:12pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/14 "2022-04-26T18:12:17Z")
This may not be the most optimal workflow. In particular, `Distributed` doesn’t offer a tree-based `reduce`, which might make `mapreduce` operations (ie. distributed for loops) seriously slow compared to MPI. It’s really hard to compare with MPI if one needs to transfer large data across cores.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [April 26, 2022, 6:20pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/15 "2022-04-26T18:20:14Z")
> [@Bruno\_Amorim](#):
>
> In the case of embarrassingly parallel problems isn’t it better to have SLURM (or whatever) launch several julia processes?
A lot of people use Slurm batch files to launch Julia processes, but I am a big fan of doing everything within Julia using `ClusterManagers` (which internally builds an `srun` command). My `addprocs` for example looks like
```julia
addprocs(SlurmManager(500), N=17, topology=:master_worker, exeflags="--project=.")
```
launching 500 worker processes across 17 nodes, after which I just use `pmap` to launch my long-running functions on each of the processes.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [April 26, 2022, 6:20pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/16 "2022-04-26T18:20:59Z")
I see, I don’t know much about tree-based reduce. For my work, `pmap` returns an array of values from each of the worker processes and I do my own reduce operations on it.
What does tree-based reduce mean?
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [April 26, 2022, 6:26pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/17 "2022-04-26T18:26:30Z")
Since we’re talking about use cases, I have to construct a 3D or 4D lattice of arbitary side length, and divide it into chunks to be processed by each worker. Each point is processed knowing its current value and the values of the nearest neighbors. In fact, in my implementation I don’t use `pmap`, I launch every process once and `@sync` them. I do this for a bunch of loops, making sure that the points each process is working on is totally independent and doesn’t require locking the array. The only exchange between processes is when neighbors happen to be on a part of the lattice owned by another process, which `DistributedArrays.jl` handles automatically for me simply indexing the global array object.
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [September 7, 2022, 2:59pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/18 "2022-09-07T14:59:28Z")
Hi @ultrapoci, I met a similar problem. I use the iterative solvers on a slurm cluster using DistributedArrays and I have to communicate data between workers every iteration, which is really slow because no InfiniBand support (10~200times slower than in-node data transfer). Do you find any way to avoid it? Or maybe we have to rework with MPI.jl?
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [September 7, 2022, 3:36pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/19 "2022-09-07T15:36:40Z")
Sadly, I’ve found no solution. I decided to port the program to Rust. Despite not having cluster support (yet), I can simply run each simulation by itself on each node manually. It’s a bit tedious, but it gets the job done in a fraction of the time Julia would take. I still use Julia from time to time for data analysis, the REPL is quite handy for this.
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [September 7, 2022, 3:51pm UTC](https://discourse.julialang.org/t/distributed-jl-distributedarray-jl-with-infiniband-cluster/80049/20 "2022-09-07T15:51:54Z")
Sadly, I don’t know about the InfiniBand thing until all the program is ok. I find the bottleneck of communication these days while optimizing the codes and then the administrator of cluster asked me whether my codes use InfiniBand to send message or not. It’s too late 😭