# Setting up distributed workers on seperate nodes of cluster - PBS and OpenMPI

**URL:** <https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013>\
**Category:** Julia at Scale\
**Created:** [May 12, 2021, 10:21am UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013 "2021-05-12T10:21:44Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 12, 2021, 10:21am UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/1 "2021-05-12T10:21:44Z")

</div>

I’m quite new to distributed computing and high performance computing so please be kind.

My university has access to an HPC which uses PBS/Torque. I’m currently running the [ProgressiveHedging.jl](https://github.com/NREL/ProgressiveHedging.jl) package using all the cores of one node, but I have a lot of scenarios and I would like to run it on multiple cores on multiple nodes.

So far I figured out that I should pass `--machine-file=$PBS_NODEFILE` when I start up Julia, but I get the following error when trying to start julia on 2 nodes with 2 cores each:

```julia
✔ [May/12 12:08] <user_and_path> $ julia --machine-file=$PBS_NODEFILE
Joining job 50738709.tier2-p-moab-2.tier2.hpc.kuleuven.be
Joining job 50738709.tier2-p-moab-2.tier2.hpc.kuleuven.be
ERROR: TaskFailedException:
Unable to read host:port string from worker. Launch command exited with error?

```

So far I’ve managed to glean this information from my sysadmin:

> The nodes communicate via infiniband network, which is a topology built on top of the hardware/power/storage setup. The using OpenMPI communicate wit one another via Torque, whereas the jobs using Intel MPI use SSH. When your account was create, an internal ssh key is also automatically generated inside your ~/.ssh folder; that is for this purpose.

So the nodes should be able to communicate and startup using SSH, so why does the above lead to an error? Also is the contents of `PBS_NODEFILE` enough for Julia to figure out what to do? (see below)

```julia
r25i13n16
r25i13n16
r25i13n17
r25i13n17

```

---

<div class="post-metadata">

**Author:** ![LeePhillips](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leephillips/32/205514_2.png) [@LeePhillips](https://discourse.julialang.org/u/LeePhillips)\
**Post date:** [May 12, 2021, 5:46pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/2 "2021-05-12T17:46:06Z")

</div>

Look at [Multi-processing and Distributed Computing · The Julia Language](https://docs.julialang.org/en/v1/manual/distributed-computing/#Starting-and-managing-worker-processes) for the format of the machine-file.

Can you reach those nodes from the shell? What happens if you type `ssh r25i13n16 hostname`?

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 13, 2021, 6:08am UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/3 "2021-05-13T06:08:57Z")

</div>

Thanks for the reference, I had missed that in the documentation. It would be good to have an actual example of a `machine-file` as well as typing it out. I will make a pull request / issue.

So I tried again, my `$PBS_NODEFILE` looks like:

```julia
r25i13n07
r25i13n07
r25i13n11
r25i13n11

```

If I type `ssh r25i13n07` or `ssh r25i13n11` it takes me to that node, so default `user` and `ports` work. If I understand it well, my machine file should then look like:

```julia
2*r25i13n07
2*r25i13n11

```

However I still get the same error, even when I make the hosts and ports explicit and drop the attempt to start 2 workers on each node:

```julia
<user>@r25i13n07:22
<user>@r25i13n11:22

```

---

<div class="post-metadata">

**Author:** ![LeePhillips](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leephillips/32/205514_2.png) [@LeePhillips](https://discourse.julialang.org/u/LeePhillips)\
**Post date:** [May 13, 2021, 2:16pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/4 "2021-05-13T14:16:44Z")

</div>

Your machine file has the right syntax.

Things to check now might be: can you ssh into the main node from the worker nodes; are all ssh connections passwordless; after ssh logging in to the worker nodes, can you run julia on them?

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 13, 2021, 4:25pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/5 "2021-05-13T16:25:45Z")

</div>

I can ssh into all of them, but I can’t start Julia. My setup is that my pbs script calls a `setup.rc` file which (among other things) adds the Julia binary to the path, but even after sourcing it through ssh I get the same error.

I noticed that even after sourcing it, if I go to that worker and try to start Julia it doesn’t work. So I think I have to somehow do run this setup script on all workers before my job starts.

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 17, 2021, 11:39am UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/6 "2021-05-17T11:39:24Z")

</div>

@LeePhillips I contacted my ICT support team and they suggested adding the workers from within Julia:

```julia
addprocs([("r26i27n06", 1)]; exename=`/data/leuven/331/vsc33168/julia-1.5.2/bin/julia`, tunnel=true)

```

This still does not work, even if I specify `user@host:port`. I even tried specifying the bind address:

```julia
addprocs([("vsc33168@r25i27n22:22 r25i27n19:22", 1)]; exename=`/data/leuven/331/vsc33168/julia-1.5.2/bin/julia`, tunnel=true)

```

No luck. I’m really at a loss here.

---

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [May 17, 2021, 1:07pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/7 "2021-05-17T13:07:38Z")

</div>

I seem to recall having a problem like that before, too. I was advised to run the same version of julia on _all_ nodes (master and workers) and I did, but I’m not sure if that was what fixed it. When you `ssh` do you set `StrictHostKeyChecking`? If I recall that was my issue. The following works for me:

```julia
    params=(exename="/home/..../julia-1.6.0/bin/julia",
            dir=remoteDir,
            tunnel=true,
            sshflags=`-o StrictHostKeyChecking=no`)
    addprocs(procLimits; params...)

```

where `procLimits` is a vector of `Tuple{String,Int}` like the first argument of your call to `addprocs()` above.

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 17, 2021, 3:21pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/8 "2021-05-17T15:21:09Z")

</div>

Nope, same problem:

```julia
addprocs([("r25i27n23", 1)]; exename=`/data/leuven/331/vsc33168/julia-1.5.2/bin/julia`, tunnel=true, sshflags=`-o StrictHostKeyChecking=no`)
Joining job 50746254.tier2-p-moab-2.tier2.hpc.kuleuven.be
ERROR: TaskFailedException:
Unable to read host:port string from worker. Launch command exited with error?

```

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 19, 2021, 9:55am UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/9 "2021-05-19T09:55:19Z")

</div>

@healyp @LeePhillips do either of you have any tips on debugging this? The error message I get from Julia is not very helpful, and I’m at a loss for going forward.

EDIT: the IT person helping me suggested I add the flags `-vv` which gave the output below (screenshot since I can’t copy paste from our service desk ticket). He also noticed that Julia 0.4.7 was able to get the workers running by simply specifying `--machinefile=$PBS_NODEFILE`, but not so for versions \>1.

 ![ssh_output](https://global.discourse-cdn.com/julialang/original/3X/4/d/4d86714130b814a8fdce8ea886bbdf8d21eb0a9f.png)

---

<div class="post-metadata">

**Author:** ![gobs](https://avatars.discourse-cdn.com/v4/letter/g/bc8723/32.png) [@gobs](https://discourse.julialang.org/u/gobs)\
**Post date:** [May 19, 2021, 4:45pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/10 "2021-05-19T16:45:08Z")

</div>

The solution ended up being very specific to the HPC I’m using. I’ve attached a screenshot below in case it is useful for someone else in the future, but it’s so particular that I somewhat doubt it:

 ![distributed_problem](https://global.discourse-cdn.com/julialang/original/3X/7/9/79cbbfaab137bb89a5e978a5296615179d3176ba.png)

---

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [May 19, 2021, 8:38pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013/11 "2021-05-19T20:38:40Z")

</div>

Happy to hear you have a way forward.

Your fix was most certainly not what did it for me, however J.