New to Julia, and also new to SLURM. I’m thinking about the following workflow for my project and would like to see if it seems a sensible strategy, especially considering the specificities of the Julia language.
I plan to run hundreds (thousands…) of completely independent numerical simulations with varying parameters, where each simulation takes anywhere between a few minutes to a few hours to complete. I have access to a SLURM high performance cluster with a very large number of CPU, yet I can request approximately 64 of them at most per task (although I can run several of these 64-CPU tasks at the same time of course) as most machines have fewer processors than this.
Given this 64 CPU limit, I am not envisioning to use exclusively Julia’s native multiprocessing modules since it would require running serially a lot of 64 simulation batches. I think SLURM task arrays (i.e., using something like #SBATCH --array=1-1000) would be better suited for the job, where my main Julia computing script could be called X times, these processes would be distributed among the nodes for a total far greater than 64 simultaneous tasks.
Does this sound like an optimal strategy given my project? Anything I have overlooked? Or are there other strategies I could consider native to Julia?
Could you give an example of how SLURM arrays work? I use SLURM quite often but through ClusterManagers.jl.
If you have a submit script like this:
myscript.sh
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --time=01:00:00
#SBATCH --array=1-30
module load julia
julia myfile.jl
and you run sbatch myscript.sh, 30 identical jobs are submitted, and each job can be identified by a SLURM_ARRAY_TASK_ID
At the top of myfile.jl you can put: id = Base.parse(Int, ENV["SLURM_ARRAY_TASK_ID"])
so that every run of your Julia file has a unique number it can use to set up parameters etc.
This is how I’ve always submitted serial jobs to the cluster. I don’t have the expertise to answer your question @tb877, but I have used array jobs to submit thousands of simulations successfully. It’s what everyone in my office does (but we are all just copying each other). I’m not sure if submitting the jobs with cpu-per-task=64 ends up with a better use of resources because you use every cpu on the node. Perhaps this has some effect on your scheduling priority over the long term. But I’ve never had any problems just using array jobs.
@affans: I have seen a few posts about ClusterManagers.jl (and read the docs) but not a lot of MWE. My workflow is pretty simple (i.e., no communication between jobs, etc.) so I guess it shouldn’t be so complicated. Would you mind sharing a few lines of code about your strategy using this module?
When you submit julia myfile.jl 30 individual times, you are launching a fresh instance and all the code has to be compiled per instance, which is not a big deal if your simulation is long-running, but still.
Here is how I use ClusterManagers to manage everything within Julia (i.e., no bash files).
using ClusterManagers
addprocs(SlurmManager(500), N=16, verbose="", topology=:master_worker, exeflags="--project=$(Base.active_project())")
This adds 500 worker processes over N = 16 nodes (my cluster has 32 cpus per node). I also pass in an argument to activate the correct environment.
Then it’s a matter of running pmap
cd = pmap(1:nsims) do x
@info "Starting simulation $x on host $(gethostname()), id: $(myid())"
run_model()
end
The variable cd is an array of each output of run_model(). If nsims = 2000, then ClusterManagers will automatically launch jobs in batches of 500 (as that’s how many workers I’ve requested). I then run my averaging scripts/plotting on cd.
This is exactly the kind of thing I was “worried” about (I even started writing along these lines in my original post but ended up not including it). My simulations are indeed usually long (several minutes to a few hours) so I think I’m okay here, but good call.
Thanks for the example code. Is there any advantage to create the SLURM array through Julia? It seems more complicated than through a shell script tbh. I suppose I’ll start with a shell script but keep this in mind when I’m more familiar with Julia, as it would eliminate the need for a shell script altogether.
I think I am confused. Do you submit one job to the scheduler, requesting 16 nodes with all 32 cpus per node? And then ClusterManagers handles what resources you have available? But you said no bash files, so you don’t have a submissoin script that requests the resources from the slurm scheduler? e.g. nodes=16, cpus-per-node=32?
Other than compilation time, is there an advantage? Does requesting all the CPUs on the node have advantages? Also if your jobs go through together, then the compilation is happening in parallel so it isn’t too bad. But also, if you have to request 16 full nodes, is this not difficult for the scheduler as you need 16 nodes to be free? Whereas if you submit as an array job, once a node becomes free your job can go through? I think I must have how this works in my head wrong.
[affans@hpc ~]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
10058 defq julia-88 affans R 1:05 15 node[001-010,012-016]
Running 10 “simulations” looks like this:
julia> pmap(1:10) do x
gethostname()
end
10-element Vector{String}:
"node002"
"node003"
"node002"
"node007"
"node005"
"node005"
"node004"
"node004"
"node005"
"node002"
where each “simulation” simply returned the node on which the function was run.
@tb877 The advantage here is that you are fully contained within julia – as your simulations finish, you can process them right away while they are in memory. But as @p_f says, my way will request all n nodes to be available. On the other hand, I don’t really have much experience with Slurm arrays but maybe I can play around with that later on.
I see! This is very interesting and good to be aware of.
Haven’t quite worked out how this would work with the cluster I’m on, as the cluster managers are constantly warning “Do not run jobs on the login node”. Obviously the actual work here isn’t being performed on the login node, but I’m not sure what their criteria is for killing code that is running on login nodes. I think using up lots of memory on the login node might be a no-no, I’m not sure.
Well if SLURM is configured properly, then worker processes that Julia launches should always be on the compute nodes. I have a similar problem where my head node has only 32gb of RAM while the compute nodes have 128gb. This means I have to be careful when all the data is sent back from the compute nodes back tot he end node (i.e., when pmap finishes running).
