# A neat Julia/SLURM trick

**URL:** <https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123>\
**Category:** General Usage\
**Created:** [May 14, 2021, 6:12am UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123 "2021-05-14T06:12:03Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [May 14, 2021, 6:12am UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/1 "2021-05-14T06:12:04Z")

</div>

This took some figuring out and I haven’t seen it mentioned for Julia, so I figured I’d share. Hope its helpful to someone!

If you often submit Julia scripts on a SLURM cluster and hate having to have a separate SLURM script _and_ Julia script file, it turns out you can do both in one. The trick is basically that the file needs to be both a valid shell script as well as a valid Julia script. Since `#= ... =#` is a Julia multiline comment but not a shell multiline comment, you can use this to execute shell code when the script is parsed as a shell script, but which is hidden from Julia so it doesn’t cause errors. It looks like e.g. this:

```julia
#!/usr/bin/env sh
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out
#=
srun julia $(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
exit
# =#

using SomePackge
do_julia_stuff()

```

Put this in a file `script.jl` and then `sbatch script.jl` and its submitted. When your job starts, it’ll execute this file as a shell script and execute the `srun` line which then `srun`s the same file again with Julia, which starts executing at the `using SomePackage` line. When its done, shell will `exit` so as not to erroneously execute the rest of the file.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 14, 2021, 6:27am UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/2 "2021-05-14T06:27:20Z")

</div>

Thanks for sharing. What is the advantage of this over a “here document”?

```julia
#!/usr/bin/env sh
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

module load julia/1.6.1 ## I have to load julia before calling julia

julia << EOF

using SomePackage
do_julia_stuff

EOF
```

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [May 14, 2021, 6:40am UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/3 "2021-05-14T06:40:23Z")

</div>

TIL about “here documents” 🙏 But does it work if you need to `srun julia` or `mpiexec julia` ? Otherwise the only minor difference is probably that syntax highlighting hides all the non-Julia code in my version, whereas there’s presumably shell commands that would screw up highlighing for the rest of the file in the here-document version.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 14, 2021, 3:13pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/4 "2021-05-14T15:13:11Z")

</div>

I wouldn’t know. On my cluster, I have a script like the above and I submit it with `sbatch myscript.sh`

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [May 14, 2021, 6:37pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/5 "2021-05-14T18:37:12Z")

</div>

That’s really neat! And your method allows for passing arguments into the Julia session. I’m looking forward to trying this! Thanks very much. – Adam

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [May 14, 2021, 6:54pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/6 "2021-05-14T18:54:34Z")

</div>

See also [How do I pass options to julia using `#!/usr/bin/env` ?](https://docs.julialang.org/en/v1/manual/faq/#How-do-I-pass-options-to-julia-using-#!/usr/bin/env?)

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [May 14, 2021, 7:02pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/7 "2021-05-14T19:02:58Z")

</div>

Ha, thanks, was thinking of asking if this is worth adding to the Julia docs somewhere, looks like it already is!

---

<div class="post-metadata">

**Author:** ![VaclavMacha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vaclavmacha/32/20825_2.png) [@VaclavMacha](https://discourse.julialang.org/u/VaclavMacha)\
**Post date:** [November 22, 2021, 10:36am UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/8 "2021-11-22T10:36:04Z")

</div>

It is possible to specify an interpreter in the SLURM batch script. In most SLURM tutorials, the shell is used as an interpreter, but it can be easily changed to Julia as follows

```julia
#!/usr/bin/env julia
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

using SomePackge
do_julia_stuff()

```

or

```julia
#!/absolute/path/to/julia/executable
#SBATCH -N 10
#SBATCH -n 8
#SBATCH -o %x-%j.out

using SomePackge
do_julia_stuff()

```

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [November 22, 2021, 8:00pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/9 "2021-11-22T20:00:16Z")

</div>

This is ok if you don’t need to `srun` or `mpiexec` your script, but if you do, I think you do need something like my original example.

---

<div class="post-metadata">

**Author:** ![Chiil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiil/32/27474_2.png) [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Post date:** [November 22, 2021, 8:07pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/10 "2021-11-22T20:07:37Z")

</div>

Just out of curiosity, why do you dislike having two files?

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [November 22, 2021, 8:15pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/11 "2021-11-22T20:15:51Z")

</div>

Just like to have everything in one place for organization. E.g. if I change the name of the Julia script I would have to update the batch script file too, etc…

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 1, 2022, 3:34pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/12 "2022-03-01T15:34:59Z")

</div>

@marius311, gonna revive this to ask how you would handle Slurm array jobs in your particular setup?

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [March 1, 2022, 6:27pm UTC](https://discourse.julialang.org/t/a-neat-julia-slurm-trick/61123/13 "2022-03-01T18:27:24Z")

</div>

I’ve never used array jobs, but on quick glance looks like you could probably just add a `#SBATCH --array=...` line and then use `ENV["SLURM_ARRAY_TASK_ID"]` from inside your Julia script as needed.