# Addprocs\_slurm not connecting to all available workers
**URL:** https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149
**Category:** Julia at Scale
**Tags:** question, parallel, cluster, distributed, high-performance
**Created:** [November 27, 2024, 9:04am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149 "2024-11-27T09:04:52Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@jewh](https://discourse.julialang.org/u/jewh)
#### Post date: [November 27, 2024, 9:04am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/1 "2024-11-27T09:04:52Z")
I’m using `Distributed` and `ClusterManagers` to run large numbers of simulations in parallel on my university HPC cluster. Scheduling and job management on the cluster is handled by Slurm.
I initialise each of the parallel processes with the following:
```julia
using Distributed, ClusterManagers
available_workers = parse(Int, ENV["SLURM_NTASKS"])
addprocs_slurm(available_workers; topology = :master_worker)
```
However my job times-out in the `addprocs_slurm` step. Specifically I schedule my job using `sbatch` and the following `.sh` file:
```julia
#!/bin/bash
# set the number of nodes.
#SBATCH --nodes=32
# set the number of CPUs required.
#SBATCH --ntasks-per-node=48
# set the amount of memory needed for each CPU.
#SBATCH --mem-per-cpu=8000
# set max wallclock time (hh:mm:ss).
#SBATCH --time=5:00:00
# set the time partition for the job.
#SBATCH --partition=short
# set name of job (AND DATE!)
#SBATCH --job-name=my_job_name
# mail alert at start, end, and abortion of execution
#SBATCH --mail-type=ALL
# send mail to this address
#SBATCH --mail-user=my_email_here
# run the application
module load Julia/1.8.2-linux-x86_64
julia requirements.jl
julia my_job.jl > my_job.log
```
`requirements.jl` is a file containing all the dependencies for my simulations, and looks like:
```julia
using Pkg
dependencies = [# list of packages used here]
Pkg.add(dependencies)
```
`my_job.jl` contains the code shown at the top of my post, followed by the code to execute my simulations.
If I open `my_job.log` it looks like:
```julia
connecting to worker 1 out of 1536
connecting to worker 2 out of 1536
# 884 lines omitted
connecting to worker 887 out of 1536
# No more lines after here
```
So it looks like `addprocs_slurm` is getting stuck somehow and my job is timing out in that step waiting for my job to connect to all the workers.
I’m kind of at a loss as to where to begin debugging this. Does anyone have any suggestions?
\*The version of Julia is 1.8.2. This specific job is part of corrections for a paper I’m working on, the previous simulations for which were also run in 1.8.2, so I’m not really willing to upgrade the version unless I can avoid it.
---
### Author:  [@jewh](https://discourse.julialang.org/u/jewh)
#### Post date: [November 28, 2024, 5:34pm UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/2 "2024-11-28T17:34:04Z")
Looks like this is an open issue with `ClusterManagers.jl`: [Slurm broken · Issue #196 · JuliaParallel/ClusterManagers.jl · GitHub](https://github.com/JuliaParallel/ClusterManagers.jl/issues/196)
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [November 28, 2024, 10:57pm UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/3 "2024-11-28T22:57:53Z")
Why are you using `sbatch`? What happens if you use `SlurmManager` directly inside julia? Something like
```julia
my_job.jl:
include("requirements.jl")
function my_long_running_function()
end
```
and create a `run.jl` file with
```julia
using ClusterManagers, Distributed
addprocs(SlurmManager(n_procs), topology = :master_worker)
@everywhere include("my_job.jl")
function run()
results = pmap(1:n_sims) do x
my_long_running_function()
end
end
```
---
### Author:  [@jewh](https://discourse.julialang.org/u/jewh)
#### Post date: [November 28, 2024, 11:23pm UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/4 "2024-11-28T23:23:31Z")
> Why are you using `sbatch`
My cluster prefers use of `salloc` and `sbatch` for requesting resources. Basically all of the computing nodes on the cluster are non-interactive so I thought that `sbatch` was more apt.
> What happens if you use `SlurmManager` directly inside julia?
I haven’t tried this, but will give it a go to see if it replicates the error. I am struggling to replicate the error to be honest (or at least get a MWE). It’s happened twice when I try to run large numbers of tasks but I can’t achieve the same error on the smaller number of CPUs available on the debugging/interactive nodes.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [November 29, 2024, 2:39am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/5 "2024-11-29T02:39:03Z")
If you use `SlurmManager` within Julia’s `addproc`, it will internally construct and run an `srun` which is the same as `sbatch` really. Try it and let us know the results.
---
### Author:  [@fabiangans](https://discourse.julialang.org/u/fabiangans)
#### Post date: [November 29, 2024, 9:37am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/6 "2024-11-29T09:37:45Z")
Alterantively for your workflow there is [GitHub - kleinhenz/SlurmClusterManager.jl: julia package for running code on slurm clusters](https://github.com/kleinhenz/SlurmClusterManager.jl) which connects workers to already allocated resources instead of allocating new ones.
---
### Author:  [@jewh](https://discourse.julialang.org/u/jewh)
#### Post date: [November 29, 2024, 11:11am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/7 "2024-11-29T11:11:07Z")
Thanks for the suggestion. I need to test it at scale but on small ntasks it seems like
```julia
using Distributed, SlurmClusterManager
addprocs(SlurmManager(); topology=:master_worker)
```
runs as expected.
---
### Author:  [@jewh](https://discourse.julialang.org/u/jewh)
#### Post date: [December 4, 2024, 7:17am UTC](https://discourse.julialang.org/t/addprocs-slurm-not-connecting-to-all-available-workers/123149/8 "2024-12-04T07:17:47Z")
Have now tested this at scale, and yes it fixes the problem