Help setting up Julia on a cluster

Specific Domains Julia at Scale
, ,
18

I want to add an update and solution to this thread.

The answer by @ElOceanografo is great, specially if you are interested in fine control with the cluster resource manager. I would say that the most clean solution nowadays is through the ClusterManagers.jl package.

Basically, the package defines various functions addprocs_slurm(), addprocs_pbs(), etc. which can be used in place of the Julia builtin addprocs(). When you start Julia, instead of asking for processes with addprocs(), you simply do:

using ClusterManagers

addprocs_pbs(100) # request 100 processes in the cluster using PBS

# REST OF SCRIPT GOES HERE

This will take care of submitting the job with the appropriate resource manager without having to write the job script manually.

2 Likes