# Distributing parallel tasks/workers over multiple nodes using SLURM
**URL:** https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574
**Category:** General Usage
**Tags:** hpc, distributed, pmap
**Created:** [May 5, 2021, 12:52pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574 "2021-05-05T12:52:51Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@schmuelinsky](https://discourse.julialang.org/u/schmuelinsky)
#### Post date: [May 5, 2021, 12:52pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/1 "2021-05-05T12:52:51Z")
I’m trying to compute a series of Monte Carlo (MC) integrations on an HPC using SLURM. Since the MC samples are all independent, my approach is to distribute them as parallel tasks using `pmap()`.
Each MC sample is quite memory-intensive, so I want to allocate each task to its own CPU (with 4GB of RAM). Each computing node has 16 CPUs, and I am aiming at 256 MC samples, so I need 16 nodes.
According to cluster support, something is not working as intended, and all of the tasks are somehow only being stuffed onto the first node, which causes them to have way too little RAM and fail.
Unfortunately, I have no idea about how to configure SLURM tasks/workers within Julia. I got a few lines of Julia code from my former supervisor and just pasted them:
```nohighlight
using ClusterManagers
const SLURM = true
N_cpus = try
parse(Int, ARGS[1])
catch
3
end
# 1 main process and $N_worker sub processes
# on the cluster add all processes
SLURM ? addprocs(SlurmManager(N_cpus)) : addprocs(N_cpus - 1)
...
pmap(...)
```
In the SLURM submit file, I specified my nodes and tasks:
```julia
#SBATCH --nodes=16
#SBATCH --ntasks-per-node=16
...
echo "------------------------------------------------------------"
echo "SLURM JOB ID: $SLURM_JOBID"
echo "$SLURM_NTASKS tasks"
echo "------------------------------------------------------------"
module load julia/1.5.3
julia --project=@. julia_file.jl $SLURM_NTASKS
```
Can someone spot my mistake?
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [May 5, 2021, 1:10pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/2 "2021-05-05T13:10:12Z")
Try
```julia
addprocs(SlurmManager(256), N=16)
```
---
### Author:  [@schmuelinsky](https://discourse.julialang.org/u/schmuelinsky)
#### Post date: [May 5, 2021, 1:18pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/3 "2021-05-05T13:18:10Z")
Instead of this whole line?
`SLURM ? addprocs(SlurmManager(N_cpus)) : addprocs(N_cpus - 1)`
Or like this?
`SLURM ? addprocs(SlurmManager(256), N=16) `
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [May 5, 2021, 1:21pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/4 "2021-05-05T13:21:40Z")
The latter. I was just trying to highlight the `N` argument in `addprocs`. It tells Slurm how many nodes to use. This should set up 16 workers over 16 nodes.
Offtopic, I will also mention that `pmap` runs on the headnode and all the results from the computations are passed back to the headnode. Often the headnode has very little memory relative to the compute nodes. For example, my headnode only has 32gb of memory compared to 256gb on each of my nodes. So just keep a track of your memory usage.
---
### Author:  [@schmuelinsky](https://discourse.julialang.org/u/schmuelinsky)
#### Post date: [May 6, 2021, 11:35am UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/5 "2021-05-06T11:35:41Z")
The results I return to the head node shouldn’t be significant in size. But thanks for the info!
I think I’m misunderstanding something about the syntax:
`SLURM ? addprocs(SlurmManager(256), N=16)`
gives me an error:
`LoadError: syntax: colon expected in "?" expression`
There is a colon in the original line, so I guess I’m missing the last part
`: addprocs(256 - 1)` or something like that?
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [May 6, 2021, 12:33pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/6 "2021-05-06T12:33:12Z")
There must be a way of dealing with the pmap being run on the head node in Slurm.
Perhaps we need to start a job on a compute node which then uses SlurmManager to add the workers.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [May 6, 2021, 4:27pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/7 "2021-05-06T16:27:18Z")
Not sure what you mean here.
```julia
SLURM ? addprocs(SlurmManager(N_cpus), N=16) : addprocs(N_cpus - 1)
```
should work fine.
---
### Author:  [@schmuelinsky](https://discourse.julialang.org/u/schmuelinsky)
#### Post date: [May 7, 2021, 3:51pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/8 "2021-05-07T15:51:00Z")
I was just confused because I mistakenly thought I should leave out the last part of the command
` : addprocs(N_cpus - 1)` which was there originally.
Anyway, I seem to have found a simpler solution:
There is a package [SlurmClusterManager.jl](https://github.com/kleinhenz/SlurmClusterManager.jl) which handles all of the CPU and node allocations automatically. That way, I only need to include the lines
```julia
using Distributed, SlurmClusterManager
addprocs(SlurmManager())
```
in my Julia script. That way, I only need to edit the `.sh` submit file if I want to allocate the tasks differently.
But thanks for your help nonetheless!
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [May 7, 2021, 4:08pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/9 "2021-05-07T16:08:12Z")
There is a difference between the two packages. The documentation for `SlurmClusterManager` says
> Requires that `SlurmManager` be created inside a Slurm allocation created by sbatch/salloc. Specifically `SLURM_JOBID` and `SLURM_NTASKS` must be defined in order to construct `SlurmManager` . This matches typical HPC workflows where resources are requested using sbatch and then used by the application code. In contrast `ClusterManagers.jl` will _dynamically_ request resources when run outside of an existing Slurm allocation.
In other words, you can either allocate nodes **outside** julia using `sbatch/srun` and manage workers externally or you can allocate everything inside Julia using `ClusterManagers`. I like the latter better since it keeps my code clean and concise and without introducing script files. The `N` argument in `addprocs(SlurmManager(N_cpus), N=16)` precisely tells the script how many nodes to allocate so it should work, which then calls `srun` to dynamically allocate nodes. No `.sh` files needed.
However, if you are more comfortable with manually running `sbatch/srun`, then yes use the `SlurmClusterManager`
---
### Author:  [@schmuelinsky](https://discourse.julialang.org/u/schmuelinsky)
#### Post date: [May 7, 2021, 4:14pm UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/10 "2021-05-07T16:14:45Z")
As I understand it, it is common practice to use a `.sh` file on our cluster since there are a lot of parameters (QOS, time limits, notifications) to define when using `sbatch`.
I don’t know if it is obligatory, but it doesn’t bother me, so I’m gonna stick with it 🙂
---
### Author:  [@ZhiguangLu](https://discourse.julialang.org/u/ZhiguangLu)
#### Post date: [October 25, 2022, 3:41am UTC](https://discourse.julialang.org/t/distributing-parallel-tasks-workers-over-multiple-nodes-using-slurm/60574/11 "2022-10-25T03:41:45Z")
If you deal with the problem of multiple-nodes, can you tell me how do you do it? Especially in your julia code, i want to know the detail of the code. Thanks!