Number of threads assigned via SLURM

Specific Domains
,
1

I am trying to run some multi-threaded code on a cluster using SLURM.

#SBATCH --cpus-per-task=4
JPATH=$HOME/.juliaup/bin
export JULIA_DEPOT_PATH="/my/storage/dir/.julia:"
export JULIA_NUM_THREADS=auto
$JPATH/julia main_0_test.jl

Using ThreadPinning.jl, I pin the threads according to affinity mask and then check threadinfo with slurm=true.

Hostname: 	r184.ib.bridges2.psc.edu
CPU(s): 	2 x AMD EPYC 7742 64-Core Processor
CPU target: 	znver2
Cores: 		128 (128 CPU-threads)
NUMA domains: 	2 (64 cores each)

SLURM: 2 assigned CPU-threads. Will only show those below.

Julia threads: 	4

CPU socket 1
  4,5


# = Julia thread, # = >1 Julia thread

(Mapping: 1 => 124, 2 => 125, 3 => 126, 4 => 127,)
Threads.nthreads() = 4

I am confused as to why it says slurm only assigned 2 cpu threads. Number of Julia threads are correct, and the mapping also shows where they have been assigned. The computations also run faster as I increase the number of cpus requested. I am wondering what does the “2 assigned CPU-threads” mean and if it will lead to collisions when running multiple jobs. Would it be better to ask these questions to the team running the clusters instead?

2

My first thought would be that the chip has hyperthreads (or whatever the AMD equivalent is), so Julia pins its 4 threads to 4 logical cores over 2 physical cores and SLURM only allocates or sees the 2 physical. But that’s a complete guess as I’m mostly unfamiliar with SLURM.

Is the program using SlurmClusterManager.jl?

3

Even when I run with more threads, I get the same slurm message. For example with 16 threads:

Hostname: 	r200.ib.bridges2.psc.edu
CPU(s): 	2 x AMD EPYC 7742 64-Core Processor
CPU target: 	znver2
Cores: 		128 (128 CPU-threads)
NUMA domains: 	2 (64 cores each)

SLURM: 2 assigned CPU-threads. Will only show those below.

Julia threads: 	16

CPU socket 1
  4,5


# = Julia thread, # = >1 Julia thread

(Mapping: 1 => 94, 2 => 95, 3 => 96, 4 => 97, 5 => 98, ...)
Threads.nthreads() = 16

I am not using SlurmCluserManager, the use case seemed more applicable towards distributed computing and I didn’t look further into it.

4

How many cpu(-thread)s are you requesting through SLURM? What does your job script look like, especially the SLURM parameters at the top of it?

5

Earlier I was asking for a certain number of cpus through cpus-per-task command. Now I am using n instead. Here is the full .sh file:

#!/bin/bash
#SBATCH --job-name=mu_0_multi_0.1_4
#SBATCH -p RM-shared
#SBATCH -N 1
#SBATCH -n 12
##SBATCH --cpus-per-task=8
#SBATCH --time=72:00:00
#SBATCH --output=%x.out
#SBATCH --error=%x.err
#SBATCH --mail-type=END,FAIL
echo "starting $SLURM_JOBID at `date` on `hostname`"

JPATH=$HOME/.juliaup/bin
echo "julia obtained from $JPATH"
export JULIA_DEPOT_PATH="/project_path/.julia:"
export JULIA_NUM_THREADS=auto

$JPATH/julia main_multi_0_1_4.jl

echo "ended at `date` on `hostname`"

exit 0

Running getcpuids(), pinthreads(:affinitymask) and then threadinfo() inside the script:

Hostname: 	r040.ib.bridges2.psc.edu
CPU(s): 	2 x AMD EPYC 7742 64-Core Processor
CPU target: 	znver2
Cores: 		128 (128 CPU-threads)
NUMA domains: 	2 (64 cores each)

SLURM: 8 assigned CPU-threads. Will only show those below.

Julia threads: 	12

CPU socket 1
  38,39,41,58,59,60,61,62


# = Julia thread, # = >1 Julia thread

(Mapping: 1 => 52, 2 => 53, 3 => 54, 4 => 55, 5 => 85, ...)

For some reason, Slurm seems to have assigned 8 cpu-threads even when I have asked for 12 threads. The ids obtained from getcpuids and the ones that get pinned also do not match. I am using JULIA_NUM_THREADS=auto, and the number of threads initialized do match the number of cpus requested.
I do not get consistent speeds with different instances of the job. Some run fast, and some slow while the architecture of each node is same and I have asked for the same number of threads.

6

This is strange. Is there some SLURM output regarding this in the job log file? I’d focus on figuring out what’s going on here before doing anything Julia-related.

This is also weird. Normally you shouldn’t be able to use CPU-threads that aren’t part of your SLURM allocation. (Again, not really a Julia issue.)

Perhaps you should talk to the Cluster admins/support team.