# Multithreading problems in running Julia in a Slurm cluster

**URL:** <https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826>\
**Category:** General Usage\
**Tags:** question, cluster, slurm\
**Created:** [April 11, 2024, 10:57am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826 "2024-04-11T10:57:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maphdze](https://avatars.discourse-cdn.com/v4/letter/m/ea5d25/32.png) [@maphdze](https://discourse.julialang.org/u/maphdze)\
**Post date:** [April 11, 2024, 10:57am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/1 "2024-04-11T10:57:26Z")

</div>

Hi! I am confused about running Julia code in a Slurm cluster.  
I am solving a nonlinear PDE so I have to solve it step by step. In each step a linear equation system Ax=b (from FEM so A is sparse) is solved by `\` which actually adopts the `SuitSparse` package. In my personal laptop and the desktop in office, the `\` is automatically multithreaded even run with ‘julia -t 1’.  
Today I try to run my code in a Slurm cluster. When I applied for 4 cores in a single node, the output file is weird. It looks like my code is running in each core independently. Attached is my script to submit the job to cluster and the output file.

```julia
#!/bin/bash
#SBATCH -J Test 
#SBATCH -p intel  
#SBATCH -N 1 
#SBATCH -n 4 
#SBATCH --ntasks-per-node=4   

hostname
srun julia -t 4 StaMmdSys.jl
sleep 100

```

The weird output file in which each incremental step and iteration step is computed several times

```julia
cpui03
Num Ste 1
Num Ste 1
Num Ste 1
Num Ste 1
  Num Ite 1
    ResErr 1.1815291149415338e-26 LoaFac 31.340781306503
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 2
  Num Ite 1
    ResErr 1.1815291149415338e-26 LoaFac 31.340781306503
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 2
  Num Ite 1
    ResErr 1.1815291149415338e-26 LoaFac 31.340781306503
    Max Dam 0.0 Min Deg 1.000000000000001
  Num Ite 1
    ResErr 5.613078015512632e-27 LoaFac 62.681562613006
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 2
Num Ste 3
  Num Ite 1
    ResErr 1.1815291149415338e-26 LoaFac 31.340781306503
    Max Dam 0.0 Min Deg 1.000000000000001
  Num Ite 1
    ResErr 5.613078015512632e-27 LoaFac 62.681562613006
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 3
Num Ste 2
  Num Ite 1
    ResErr 3.098702727267662e-7 LoaFac 94.022343919509
    Max Dam 0.0008378798567051027 Min Deg 0.983676759929105
Num Ste 4
  Num Ite 1
    ResErr 3.098702727267662e-7 LoaFac 94.022343919509
    Max Dam 0.0008378798567051027 Min Deg 0.983676759929105
Num Ste 4
  Num Ite 1
    ResErr 1.565185266046758e-5 LoaFac 125.36136059678219
    Max Dam 0.010911503021434944 Min Deg 0.8866271918773956
  Num Ite 1
    ResErr 5.613078015512632e-27 LoaFac 62.681562613006
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 3
  Num Ite 1
    ResErr 5.613078015512632e-27 LoaFac 62.681562613006
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 3
  Num Ite 1
    ResErr 1.565185266046758e-5 LoaFac 125.36136059678219
    Max Dam 0.010911503021434944 Min Deg 0.8866271918773956
  Num Ite 1
    ResErr 3.098702727267662e-7 LoaFac 94.022343919509
    Max Dam 0.0008378798567051027 Min Deg 0.983676759929105
  Num Ite 1
    ResErr 3.098702727267662e-7 LoaFac 94.022343919509
    Max Dam 0.0008378798567051027 Min Deg 0.983676759929105
  Num Ite 2
    ResErr 1.645042974830353e-7 LoaFac 125.17614239795542
    Max Dam 0.01105091853894029 Min Deg 0.8842490683695091
Num Ste 4
Num Ste 4
  Num Ite 2
    ResErr 1.645042974830353e-7 LoaFac 125.17614239795542
    Max Dam 0.01105091853894029 Min Deg 0.8842490683695091
Num Ste 5
  Num Ite 1
    ResErr 2.9130673485754505e-5 LoaFac 156.467617106916
    Max Dam 0.04471510355125102 Min Deg 0.7560656522648532
  Num Ite 1
    ResErr 1.565185266046758e-5 LoaFac 125.36136059678219
    Max Dam 0.010911503021434944 Min Deg 0.8866271918773956
Num Ste 5
  Num Ite 2
    ResErr 1.645042974830353e-7 LoaFac 125.17614239795542
    Max Dam 0.01105091853894029 Min Deg 0.8842490683695091
Num Ste 5
  Num Ite 1
    ResErr 1.565185266046758e-5 LoaFac 125.36136059678219
    Max Dam 0.010911503021434944 Min Deg 0.8866271918773956
  Num Ite 1
    ResErr 2.9130673485754505e-5 LoaFac 156.467617106916
    Max Dam 0.04471510355125102 Min Deg 0.7560656522648532
  Num Ite 2
    ResErr 3.927810503996626e-6 LoaFac 155.8450166806799
    Max Dam 0.056704006161101106 Min Deg 0.7223674694734912
  Num Ite 2
    ResErr 3.927810503996626e-6 LoaFac 155.8450166806799
    Max Dam 0.056704006161101106 Min Deg 0.7223674694734912
  Num Ite 3
    ResErr 5.410640813519663e-7 LoaFac 155.657378580163
    Max Dam 0.060797662055178815 Min Deg 0.7137250018025755

```

A normal output file should look like this (get it from cluster with only 1 core)

```julia
cpui03
Num Ste 1
  Num Ite 1
    ResErr 1.1815291149415338e-26 LoaFac 31.340781306503
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 2
  Num Ite 1
    ResErr 5.613078015512632e-27 LoaFac 62.681562613006
    Max Dam 0.0 Min Deg 1.000000000000001
Num Ste 3
  Num Ite 1
    ResErr 3.098702727267662e-7 LoaFac 94.022343919509
    Max Dam 0.0008378798567051027 Min Deg 0.983676759929105
Num Ste 4
  Num Ite 1
    ResErr 1.565185266046758e-5 LoaFac 125.36136059678219
    Max Dam 0.010911503021434944 Min Deg 0.8866271918773956
  Num Ite 2
    ResErr 1.645042974830353e-7 LoaFac 125.17614239795542
    Max Dam 0.01105091853894029 Min Deg 0.8842490683695091
Num Ste 5
  Num Ite 1
    ResErr 2.9130673485754505e-5 LoaFac 156.467617106916
    Max Dam 0.04471510355125102 Min Deg 0.7560656522648532
  Num Ite 2
    ResErr 3.927810503996626e-6 LoaFac 155.8450166806799
    Max Dam 0.056704006161101106 Min Deg 0.7223674694734912
  Num Ite 3
    ResErr 5.410640813519663e-7 LoaFac 155.657378580163
    Max Dam 0.060797662055178815 Min Deg 0.7137250018025755
Num Ste 6
  Num Ite 1
    ResErr 7.935999964334881e-5 LoaFac 186.77680857130252
    Max Dam 0.1238986630949555 Min Deg 0.5772076022373513
  Num Ite 2
    ResErr 1.1999112834906476e-5 LoaFac 185.54765605633034
    Max Dam 0.15347340755070013 Min Deg 0.5289334799119432
  Num Ite 3
    ResErr 1.5354390771604877e-6 LoaFac 185.0418806205562
    Max Dam 0.16195976821638491 Min Deg 0.5141634286765151
  Num Ite 4
    ResErr 4.0455113498668363e-7 LoaFac 184.83622847734313
    Max Dam 0.16516550863977345 Min Deg 0.5067114390938846

```

I wonder that do I have to modify my code to run it properly in a slurm cluser? Or I should submit the job to cluster in a different manner?

---

<div class="post-metadata">

**Author:** ![kopperud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kopperud/32/208526_2.png) [@kopperud](https://discourse.julialang.org/u/kopperud)\
**Post date:** [April 16, 2024, 12:41pm UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/2 "2024-04-16T12:41:41Z")

</div>

This is how I’ve set up my slurm scripts

```julia
#SBATCH --mail-type=END,FAIL
#SBATCH --mem-per-cpu=4GB
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4

julia --threads ${SLURM_CPUS_PER_TASK} scripts/my_script.jl

```

and then `scripts/my_script.jl` would be my julia program that uses several threads. It looks like mine has 1 task and 4 cpus per task, while yours has 4 tasks. Not sure if that is a problem

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [April 16, 2024, 12:45pm UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/3 "2024-04-16T12:45:37Z")

</div>

My slurm scripts look similar to @kopperud, if that is of any help

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [April 16, 2024, 1:00pm UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/4 "2024-04-16T13:00:42Z")

</div>

> [@maphdze](#):
>
> In my personal laptop and the desktop in office, the `\` is automatically multithreaded even run with ‘julia -t 1’.

BLAS is multithreading and has separate controls. You might want to do something like this when attempting to multithread yourself.

```julia
using LinearAlgebra
BLAS.set_num_threads(1)

```

There is also an environment variable, but I forget how to spell it.

---

<div class="post-metadata">

**Author:** ![maphdze](https://avatars.discourse-cdn.com/v4/letter/m/ea5d25/32.png) [@maphdze](https://discourse.julialang.org/u/maphdze)\
**Post date:** [April 17, 2024, 2:04am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/5 "2024-04-17T02:04:56Z")

</div>

Thanks for your reply! It works for me:)

---

<div class="post-metadata">

**Author:** ![maphdze](https://avatars.discourse-cdn.com/v4/letter/m/ea5d25/32.png) [@maphdze](https://discourse.julialang.org/u/maphdze)\
**Post date:** [April 17, 2024, 2:05am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/6 "2024-04-17T02:05:09Z")

</div>

Yes it works!

---

<div class="post-metadata">

**Author:** ![maphdze](https://avatars.discourse-cdn.com/v4/letter/m/ea5d25/32.png) [@maphdze](https://discourse.julialang.org/u/maphdze)\
**Post date:** [April 17, 2024, 2:07am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/7 "2024-04-17T02:07:45Z")

</div>

> [@mkitti](#):
>
> BLAS is multithreading and has separate controls. You might want to do something like this when attempting to multithread yourself.

Does BLAS also work for `SparseCSC`? I used to think that solving the sparse linear system is controlled by `SuitSparse`.

---

<div class="post-metadata">

**Author:** ![Sevi](https://avatars.discourse-cdn.com/v4/letter/s/c67d28/32.png) [@Sevi](https://discourse.julialang.org/u/Sevi)\
**Post date:** [April 17, 2024, 8:13pm UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/8 "2024-04-17T20:13:07Z")

</div>

The question has already been answered, but since this also bit me in the past, I just wanted to add some background, which might help others. The original weird output is related to the combination of `--ntasks-per-node=4` and the `srun` command in the original script.

If you tell `sbatch` to allocate 4 tasks, it will make sure you can run up to 4 “things” (for lack of a better word, but it’s what is called “process” or “job” in other contexts) at the same time on the allocated node(s). The “tasks” are not (directly) the number of available threads or cores, these are called CPUs by Slurm. But each task will require at least one CPU (you can set the number with `--cpus-per-task`).

`srun` then runs job steps and uses as default for the number of tasks the same value that you specify in the script. So the single call to `srun` in the original script does simply do the same thing four times in parallel (e.g. `srun echo "hello"` prints four "hello"s if you have `--ntasks=4` or four "hello"s for each allocated node if you have `--ntasks-per-node=4`).

You could specify the number of tasks for each srun command manually, e.g.

```julia
...
#SBATCH --ntasks=4`
...

# this will do the computation only once and have `--cpus-per-task` many threads available
srun -n 1 julia my_script.jl

# or if you want to start four independent calculations in the background (in parallel) and wait for them to finish
srun -n 1 julia my_script1.jl &
srun -n 1 julia my_script2.jl &
srun -n 1 julia my_script3.jl &
srun -n 1 julia my_script4.jl &
wait

```

Some useful links and excerpts of the Slurm docs:

- [slurm - How do the terms "job", "task", and "step" relate to each other? - Stack Overflow](https://stackoverflow.com/a/46532581)
- [bash - What does the --ntasks or -n tasks does in SLURM? - Stack Overflow](https://stackoverflow.com/a/53759961)
- [Slurm Workload Manager - sbatch](https://slurm.schedmd.com/sbatch.html#OPT_ntasks)
- [Slurm Workload Manager - sbatch](https://slurm.schedmd.com/sbatch.html#OPT_cpus-per-task)

---

<div class="post-metadata">

**Author:** ![maphdze](https://avatars.discourse-cdn.com/v4/letter/m/ea5d25/32.png) [@maphdze](https://discourse.julialang.org/u/maphdze)\
**Post date:** [April 18, 2024, 7:34am UTC](https://discourse.julialang.org/t/multithreading-problems-in-running-julia-in-a-slurm-cluster/112826/9 "2024-04-18T07:34:03Z")

</div>

Thanks for your clear explanation! It is indeed helpful to me 😀