A neat Julia/SLURM trick

General Usage
1

This took some figuring out and I haven’t seen it mentioned for Julia, so I figured I’d share. Hope its helpful to someone!

If you often submit Julia scripts on a SLURM cluster and hate having to have a separate SLURM script and Julia script file, it turns out you can do both in one. The trick is basically that the file needs to be both a valid shell script as well as a valid Julia script. Since #= ... =# is a Julia multiline comment but not a shell multiline comment, you can use this to execute shell code when the script is parsed as a shell script, but which is hidden from Julia so it doesn’t cause errors. It looks like e.g. this:

#!/usr/bin/env sh
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out
#=
srun julia $(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
exit
# =#

using SomePackge
do_julia_stuff()

Put this in a file script.jl and then sbatch script.jl and its submitted. When your job starts, it’ll execute this file as a shell script and execute the srun line which then sruns the same file again with Julia, which starts executing at the using SomePackage line. When its done, shell will exit so as not to erroneously execute the rest of the file.

37 Likes
2

Thanks for sharing. What is the advantage of this over a “here document”?

#!/usr/bin/env sh
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

module load julia/1.6.1 ## I have to load julia before calling julia

julia << EOF

using SomePackage
do_julia_stuff

EOF
7 Likes
3

TIL about “here documents” :pray: But does it work if you need to srun julia or mpiexec julia ? Otherwise the only minor difference is probably that syntax highlighting hides all the non-Julia code in my version, whereas there’s presumably shell commands that would screw up highlighing for the rest of the file in the here-document version.

4

I wouldn’t know. On my cluster, I have a script like the above and I submit it with sbatch myscript.sh

5

That’s really neat! And your method allows for passing arguments into the Julia session. I’m looking forward to trying this! Thanks very much. – Adam

1 Like
6

See also How do I pass options to julia using #!/usr/bin/env ?

4 Likes
7

Ha, thanks, was thinking of asking if this is worth adding to the Julia docs somewhere, looks like it already is!

1 Like
8

It is possible to specify an interpreter in the SLURM batch script. In most SLURM tutorials, the shell is used as an interpreter, but it can be easily changed to Julia as follows

#!/usr/bin/env julia
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

using SomePackge
do_julia_stuff()

or

#!/absolute/path/to/julia/executable
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

using SomePackge
do_julia_stuff()
4 Likes
9

This is ok if you don’t need to srun or mpiexec your script, but if you do, I think you do need something like my original example.

10

Just out of curiosity, why do you dislike having two files?

11

Just like to have everything in one place for organization. E.g. if I change the name of the Julia script I would have to update the batch script file too, etc…

12

@marius311, gonna revive this to ask how you would handle Slurm array jobs in your particular setup?

13

I’ve never used array jobs, but on quick glance looks like you could probably just add a #SBATCH --array=... line and then use ENV["SLURM_ARRAY_TASK_ID"] from inside your Julia script as needed.