# A good way of running Julia MPI jobs with slurm on HPC platforms

**URL:** <https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244>\
**Category:** General Usage\
**Created:** [June 5, 2024, 4:09pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244 "2024-06-05T16:09:07Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 5, 2024, 4:09pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/1 "2024-06-05T16:09:07Z")

</div>

For some reason the Julia MPI jobs that I submit end in error due  
to a lack of instantiation / activation. I do run it as

```julia
mpiexec julia --project=~/a64fx/FinEtoolsDDParallel.jl/examples ~/a64fx/FinEtoolsDDParallel.jl/examples/heat/Poisson2D_cg_mpi_driver.jl

```

and when I do just `julia --project ~/a64fx/FinEtoolsDDParallel.jl/examples/` it works fine.

How do you guys run Julia on HPC?

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [June 5, 2024, 8:07pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/2 "2024-06-05T20:07:35Z")

</div>

For me Distributed.jl was always enough. But I only had embarassingly parallel problems anyways and MPI always looked too complicated for that 😅

---

<div class="post-metadata">

**Author:** ![xzackli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xzackli/32/38301_2.png) [@xzackli](https://discourse.julialang.org/u/xzackli)\
**Post date:** [June 5, 2024, 8:48pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/3 "2024-06-05T20:48:56Z")

</div>

On NERSC, one of my recent SLURM scripts looks like this (I cd to the project directory but one could replace the `--project=.` with a specific path)

```bash
#!/bin/bash
#SBATCH -A mp107d
#SBATCH --qos=regular
#SBATCH -C cpu
#SBATCH -t 1:00:00
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=16
#SBATCH -J big_filter
#SBATCH --exclusive

cd YOUR_PROJECT_DIR
module load cray-mpich
module load cray-hdf5-parallel
export JULIA_NUM_THREADS=8
which julia

julia --project=. -e \
    'using Pkg; using InteractiveUtils;
     Pkg.instantiate(); Pkg.precompile(); Pkg.status(); versioninfo();
     using MPI; println("MPI: ", MPI.identify_implementation());'

mpiexecjl --project=. --cpu-bind=cores julia fft_filter_6144.jl

```

---

<div class="post-metadata">

**Author:** ![lxvm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lxvm/32/50010_2.png) [@lxvm](https://discourse.julialang.org/u/lxvm)\
**Post date:** [June 5, 2024, 9:19pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/4 "2024-06-05T21:19:57Z")

</div>

On my university cluster I need to set some environment variables/flags to get a multi-node MPI setup to work. Here is a template for the slurm script I use for distributed training of a neural network.

```julia
#SBATCH --nodes=2
#SBATCH --ntasks=80
#SBATCH --cpus-per-task=1
#SBATCH --exclusive
#SBATCH --time=4:00:00
#SBATCH --mem=0

module load julia/1.9.2
module load mpi/openmpi-4.1.5

...

~/.julia/bin/mpiexecjl --project=$PROJECT --mca btl_tcp_if_include eth0 -n 80 julia --project=$PROJECT script.jl

```

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 6, 2024, 2:10pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/5 "2024-06-06T14:10:26Z")

</div>

Thank you all who responded. The setup was weird in some way: removed the depot, things changed. Not for (much) better, though.

Now all the processes try to precompile and it is a huge mess of stale lock files and such. How do you handle that?

I would like to have a single depot on the lustre file system, and each process to use that depot. So now I need to prevent each process from trying to precompile stuff…

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 6, 2024, 2:42pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/6 "2024-06-06T14:42:47Z")

</div>

So you have a separate Julia depot on all nodes?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 6, 2024, 2:43pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/7 "2024-06-06T14:43:28Z")

</div>

Using MPI is a pain! Could not agree more.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 6, 2024, 4:53pm UTC](https://discourse.julialang.org/t/a-good-way-of-running-julia-mpi-jobs-with-slurm-on-hpc-platforms/115244/8 "2024-06-06T16:53:07Z")

</div>

I figured out one way in which to run the sim with MPI, only to be stopped by a weird error: [Ookami: MPI error opal\_libevent2022\_evthread\_use\_pthreads · Issue #835 · JuliaParallel/MPI.jl · GitHub](https://github.com/JuliaParallel/MPI.jl/issues/835)

Edit: Looks like there is a solution.