# Julia on cluster, only MPI transport allowed

**URL:** <https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958>\
**Category:** General Usage\
**Tags:** parallel, distributed\
**Created:** [October 8, 2020, 12:17am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958 "2020-10-08T00:17:30Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sparrowhawk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sparrowhawk/32/9148_2.png) [@sparrowhawk](https://discourse.julialang.org/u/sparrowhawk)\
**Post date:** [October 8, 2020, 12:17am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/1 "2020-10-08T00:17:30Z")

</div>

Hi all,

I’m on a PBS cluster where I cannot simply provide a machine file to julia. After a lot of trials, I found that only MPI transport works on the cluster, and I was able to adapt an MPIClusterManagers.jl [example](https://github.com/JuliaParallel/MPIClusterManagers.jl#mpimanager-mpi-transport---all-processes-execute-mpi-code) with `MPI_TRANSPORT_ALL` on our cluster.

However, I’ve come across a rather strange phenomenon: If I request PBS for 4 cpus, and start my MPI job as

`mpirun -np 4 julia myscript.jl`

where `myscript.jl` contains the following MWE

```julia
using MPIClusterManagers, Distributed
import MPI 
MPI.Init()
rank = MPI.Comm_rank(MPI.COMM_WORLD)
size = MPI.Comm_size(MPI.COMM_WORLD)
manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL)
@info "workers are $(workers())"
rmprocs(workers())
MPIClusterManagers.stop_main_loop(manager)
exit()

```

I get `[Info: workers are [2,3,4]`

I don’t get to use the 4th cpu! Of course, if I ask PBS for 5 cpus, I get to use 4. So I’m paying for `1/ncpus_requested` more cpu time with every PBS request! `addprocs(1)` simply oversubscribes without using the last CPU.

Any ideas on how to fix this?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 8, 2020, 12:52am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/2 "2020-10-08T00:52:52Z")

</div>

What is `size`? Do you get communicator size equal to the number of requested processes? Recall that `workers` returns one less than the number of processes (i.e. minus the main one).

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [October 8, 2020, 6:53am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/3 "2020-10-08T06:53:23Z")

</div>

You can try submitting a 4 CPU job but running with:

```julia
mpirun --oversubscribe -np 5

```

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [October 8, 2020, 9:24am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/4 "2020-10-08T09:24:13Z")

</div>

I think that this is normal for MPIManager. From [GitHub - JuliaParallel/MPIClusterManagers.jl: Julia parallel constructs over MPI](https://github.com/JuliaParallel/MPIClusterManagers.jl), one of the modes executes code only on the workers:  
"

### MPIManager: only workers execute MPI code

An example is provided in `examples/juliacman.jl` . The julia master process is NOT part of the MPI cluster. The main script should be launched directly, `MPIManager` internally calls `mpirun` to launch julia/MPI workers. All the workers started via `MPIManager` will be part of the MPI cluster."

---

<div class="post-metadata">

**Author:** ![sparrowhawk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sparrowhawk/32/9148_2.png) [@sparrowhawk](https://discourse.julialang.org/u/sparrowhawk)\
**Post date:** [October 8, 2020, 1:05pm UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/5 "2020-10-08T13:05:31Z")

</div>

@mcreel, that’s almost the right answer – you sent me down the right path. I wasn’t using the paradigm you [linked](https://github.com/JuliaParallel/MPIClusterManagers.jl#mpimanager-only-workers-execute-mpi-code), where only workers use MPI code. I was using instead, the one where the master and workers all use MPI through [TCP/IP transport](https://github.com/JuliaParallel/MPIClusterManagers.jl#mpimanager-tcpip-transport---all-processes-execute-mpi-code).

When I switched to workers-only-use MPI, this MWE produces results like I expected:

```julia
using MPIClusterManagers, Distributed
manager = MPIManager(np=4)
addprocs(manager)
@info "workers are $(workers())"
exit()

```

I get `[Info: workers are [2,3,4,5]`

Please note that this is NOT run through `mpirun`. I simply ran this, after requesting PBS for 4 cpus, and then doing  
`julia myscript.jl >& outfile `

My parallel Julia code with `@sync` `@async` and `remotecall_fetch` worked as expected with near 90% CPU utilisation on all 4 requested cores on the PBS assigned node,

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 8, 2020, 1:24pm UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/6 "2020-10-08T13:24:21Z")

</div>

Hi Sparrowhawk. Can you say more about what you mean by you cannot supply a machine file?

Is this related to the MPI job launch mechanism - are you using an authentication method called munge?

---

<div class="post-metadata">

**Author:** ![sparrowhawk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sparrowhawk/32/9148_2.png) [@sparrowhawk](https://discourse.julialang.org/u/sparrowhawk)\
**Post date:** [October 10, 2020, 3:21am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/7 "2020-10-10T03:21:07Z")

</div>

Hi @johnh, I’m not sure what the authentication method is. However, if I log in with an interactive PBS job using `qsub -I`, and try specifying a machine file like so I get an error:

```julia
julia --machine-file=$PBS_NODEFILE
Host key verification failed.

```

Though the PBS\_NODEFILE exists, and I can `echo $PBS_NODEFILE`  
An admin on our cluster told me that only MPI communication is allowed between nodes, which is why I resort to using `MPIClusterManagers.jl`

Cheers

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 10, 2020, 7:44am UTC](https://discourse.julialang.org/t/julia-on-cluster-only-mpi-transport-allowed/47958/8 "2020-10-10T07:44:16Z")

</div>

It does sound like Munge authentication is being used here, but I may be on the wrong track.