# I am unable to run a simple distributed.jl code on my slurm cluster
**URL:** https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892
**Category:** Julia at Scale
**Tags:** parallel, distributed, slurm
**Created:** [February 7, 2024, 6:15pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892 "2024-02-07T18:15:52Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Shashank](https://discourse.julialang.org/u/Shashank)
#### Post date: [February 7, 2024, 6:15pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/1 "2024-02-07T18:15:52Z")
I am trying to run a simple file that is pasted below:
```julia
using Distributed
addprocs(4)
println("Number of processes: ", nprocs())
println("Number of workers: ", nworkers())
@sync @distributed for i in 1:4
sleep(1)
id, pid, host = myid(), getpid(), gethostname()
println(id, " " , pid, " ", host)
end
```
The slurm script I am using to submit the job is pasted below:
```julia
#!/bin/bash
#
#SBATCH --nodes=4
#SBATCH --partition=astro_devel
#SBATCH --ntasks-per-node=1
#SBATCH --time=0-01:00:00
#SBATCH --cpus-per-task=10
module load astro
module load intel
module load mpi/mpich-x86_64
srun /groups/astro/shashank/Julia/julia-1.10.0/bin/julia --project test.jl > test.txt
```
I would have expected the job to be distributed over 4 nodes but instead, 4 processes are created on each node. The output file is pasted below:
```julia
Number of processes: 5
Number of processes: 5
Number of workers: 4
Number of workers: 4
Number of processes: 5
Number of workers: 4
Number of processes: 5
Number of workers: 4
From worker 5: 5 16450 node764.cluster
From worker 3: 3 16448 node764.cluster
From worker 4: 4 16449 node764.cluster
From worker 2: 2 16446 node764.cluster
From worker 3: 3 15963 node765.cluster
From worker 4: 4 15964 node765.cluster
From worker 5: 5 15965 node765.cluster
From worker 2: 2 15961 node765.cluster
From worker 4: 4 23528 node762.cluster
From worker 5: 5 23529 node762.cluster
From worker 3: 3 23527 node762.cluster
From worker 2: 2 23525 node762.cluster
From worker 3: 3 19785 node763.cluster
From worker 5: 5 19787 node763.cluster
From worker 4: 4 19786 node763.cluster
From worker 2: 2 19783 node763.cluster
```
Can someone help me fix the problem? I would have expected the code to run each iteration to be run on a separate node on the same for loop on all nodes. What am I doing wrong here?
---
### Author:  [@mbauman](https://discourse.julialang.org/u/mbauman)
#### Post date: [February 7, 2024, 7:32pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/2 "2024-02-07T19:32:42Z")
You’re telling Slurm to run that Julia script on 4 nodes. And that Julia script starts 4 workers. So you end up with 16 workers.
You need to use a ClusterManager in order to get Slurm and Distributed working together.
> **[GitHub - JuliaParallel/ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl?tab=readme-ov-file#clustermanagers)**
>
> Contribute to JuliaParallel/ClusterManagers.jl development by creating an account on GitHub.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [February 7, 2024, 7:49pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/3 "2024-02-07T19:49:45Z")
Take a look at my [reply](https://discourse.julialang.org/t/distributed-computing-with-slurm-and-julia/76181/6) here as well as the entire thread.
---
### Author:  [@Shashank](https://discourse.julialang.org/u/Shashank)
#### Post date: [February 7, 2024, 7:55pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/4 "2024-02-07T19:55:57Z")
Thanks a lot both of you. I tried it as follows:
```julia
using Distributed
using ClusterManagers
addprocs(SlurmManager(4))
@sync @distributed for i in 1:4
sleep(1)
id, pid, host = myid(), getpid(), gethostname()
println(id, " " , pid, " ", host)
end
```
I get the following error:
```julia
ERROR: LoadError: TaskFailedException
nested task error: IOError: connect: host is unreachable (EHOSTUNREACH)
Stacktrace:
[1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
@ Distributed /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:1093
[2] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
@ Distributed /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:1090 [inlined]
[3] remote_do
@ /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:557 [inlined]
[4] kill
@ /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/managers.jl:726 [inlined]
[5] create_worker(manager::SlurmManager, wconfig::WorkerConfig)
@ Distributed /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:604
[6] setup_launched_worker(manager::SlurmManager, wconfig::WorkerConfig, launched_q::Vector{Int64})
@ Distributed /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:545
[7] (::Distributed.var"#45#48"{SlurmManager, Vector{Int64}, WorkerConfig})()
@ Distributed /lustre/hpc/astro/shashank/Julia/julia-1.10.0/share/julia/stdlib/v1.10/Distributed/src/cluster.jl:501
```
Is this a problem with the configuration of the cluster I am using?
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [February 7, 2024, 8:18pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/5 "2024-02-07T20:18:54Z")
That certainly sounds like a config problem. Try just `addprocs(SlurmManager(4))` and also take a look at the arguments for `SlurmManager()` to make sure you are requesting nodes/cpu.
---
### Author:  [@cnrrobertson](https://discourse.julialang.org/u/cnrrobertson)
#### Post date: [February 7, 2024, 8:29pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/6 "2024-02-07T20:29:47Z")
I’ve had a similar issue with `SlurmManager` but have not had enough time to explore/diagnose it for bug reporting.
However, I’ve found that manually creating adding worker processes on the remote nodes of the cluster has worked well:
```julia
using Distributed
# Extract all the hostname info from slurm job
node_range = ENV["SLURM_JOB_NODELIST"]
tasks_per_node = parse(Int64,split(ENV["SLURM_TASKS_PER_NODE"],'(')[1])
node_nums = parse.(Int64,filter.(isdigit, split(node_range,"-")))
nodes = [("kn$num",tasks_per_node) for num in range(node_nums...)]
addprocs(nodes)
```
You may need to check the format of the `SLURM_JOB_NODELIST` and `SLURM_TASKS_PER_NODE` variables on your cluster to make sure everything parses correctly. In my case, `SLURM_JOB_NODELIST` provides a name like `kn[15-21]` to show that I have been allocated nodes `kn15` through `kn21`.
I should also mention that I don’t use `srun` but just submit the script with `sbatch`.
---
### Author:  [@Shashank](https://discourse.julialang.org/u/Shashank)
#### Post date: [February 7, 2024, 8:36pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/7 "2024-02-07T20:36:56Z")
Thanks a lot but it seems there is a problem with my cluster. I get the following error:
```julia
Host key verification failed.^M
Host key verification failed.^M
Host key verification failed.^M
Host key verification failed.^M
ERROR: LoadError: TaskFailedException
nested task error: Unable to read host:port string from worker. Launch command exited with error?
Stacktrace:
```
It seems I will need to as the cluster admin about this.
---
### Author:  [@cnrrobertson](https://discourse.julialang.org/u/cnrrobertson)
#### Post date: [February 7, 2024, 8:44pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/8 "2024-02-07T20:44:35Z")
Ah yeah, you will need to have ssh access from the login node to the remote nodes, however that needs to be setup on your cluster.
---
### Author:  [@Shashank](https://discourse.julialang.org/u/Shashank)
#### Post date: [February 7, 2024, 9:17pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/9 "2024-02-07T21:17:44Z")
I am not sure it is possible on my cluster. Is there an alternative?
---
### Author:  [@raminammour](https://discourse.julialang.org/u/raminammour)
#### Post date: [February 7, 2024, 9:48pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/10 "2024-02-07T21:48:14Z")
Try this:
> **[GitHub - kleinhenz/SlurmClusterManager.jl: julia package for running code on...](https://github.com/kleinhenz/SlurmClusterManager.jl)**
>
> julia package for running code on slurm clusters. Contribute to kleinhenz/SlurmClusterManager.jl development by creating an account on GitHub.
Note that the script is launched from the head/dev node through `sbatch`:
```julia
sbatch -N 2 --ntasks-per-node=64 script.jl
```
And then it acquired the resources when launched by slurm:
```julia
using Distributed, SlurmClusterManager
addprocs(SlurmManager())
```
Of course this assumes that `julia` is in your `$PATH` so you can launch it.
So to be a bit more explicit: delete the `srun` in your `sbatch` script above. Replace `addprocs(4)` with `addprocs(SlurmManager())`.
Best of luck…
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [February 8, 2024, 3:19am UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/11 "2024-02-08T03:19:30Z")
Also check if [GitHub - jishnub/SlurmAddAllocatedProcs.jl: Julia package to easily add workers while using Slurm in batch mode](https://github.com/jishnub/SlurmAddAllocatedProcs.jl) helps. I had written this package when I was struggling with Slurm allocations.
---
### Author:  [@Shashank](https://discourse.julialang.org/u/Shashank)
#### Post date: [February 10, 2024, 4:30pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892/12 "2024-02-10T16:30:36Z")
Thanks a lot. The method you suggested worked with some additional changes. I had to add the following to ~/.ssh/config
```julia
Host node*
StrictHostKeyChecking no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
IdentityFile /groups/astro/shashank/.ssh/id_rsa
```
And in addition, it works only if the tunnel is set to true on my cluster for some reason.
```julia
addprocs(nodes,tunnel=true)
```
And I also had to add the public key of the frontend to authorized\_keys in ~/.ssh. This is not necessary when the connection is established by MPI.jl but is required by Distributed.jl for some reason.