# Does anyone run Singularity images across many Nodes?

**URL:** https://discourse.julialang.org/t/does-anyone-run-singularity-images-across-many-nodes/60470
**Category:** Julia at Scale
**Created:** [May 3, 2021, 3:45pm UTC](https://discourse.julialang.org/t/does-anyone-run-singularity-images-across-many-nodes/60470 "2021-05-03T15:45:17Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)
#### Post date: [May 3, 2021, 3:45pm UTC](https://discourse.julialang.org/t/does-anyone-run-singularity-images-across-many-nodes/60470/1 "2021-05-03T15:45:17Z")

</div>

As I point out in [here](https://github.com/JuliaParallel/ClusterManagers.jl/issues/164), I wish to connect ClusterManagers.jl using Singularity images, but as @kescobo says, the problem seems to be Singularity or my Cluster configuration.

Does anyone already have experience running addprocs() + Singularity across nodes to give me a complete example ? Therefore I can test where is problem is/should be.

Thanks ❤

---

<div class="post-metadata">

### Author: ![vmmhep](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vmmhep/32/34987_2.png) [@vmmhep](https://discourse.julialang.org/u/vmmhep)
#### Post date: [May 4, 2022, 9:10pm UTC](https://discourse.julialang.org/t/does-anyone-run-singularity-images-across-many-nodes/60470/2 "2022-05-04T21:10:41Z")

</div>

Did you happen to get to the bottom of this? I’m now trying the same thing on our HPC and indeed it doesn’t seem to work across nodes 🤔

---

<div class="post-metadata">

### Author: ![Noel\_Araujo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/noel_araujo/32/4862_2.png) [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)
#### Post date: [May 4, 2022, 10:07pm UTC](https://discourse.julialang.org/t/does-anyone-run-singularity-images-across-many-nodes/60470/3 "2022-05-04T22:07:04Z")

</div>

Hi, i solved in the hard way: trial and error.

I used `MPIClusterManagers`, with the option `MPI_TRANSPORT_ALL`

```julia
using Distributed , MPIClusterManagers
manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL)

# your code here

```

and my batch script is something like this:

```julia
#SBATCH --nodes=3 # node count
#SBATCH --ntasks-per-node=2
(...)
srun --mpi=pmi2 singularity run --bind=/scratch:/scratch \
    --bind=/var/spool/slurm:/var/spool/slurm \
    /home/user/folder1/work.simg /opt/julia/bin/julia -t 4 ~/folder1/folder2/example.jl

```

It is crucial to write the complete path for the singularity image and executed julia file.

If you need, i may try to write some minimal working example.