Useful Julia script tips? (workaround shebang)

General Usage
#1

I found a useful little hack so I share it here. It may already be discovered elsewhere, but it’s fun to re-discover a nice(?) trick!

If you ever tried to write a julia script and put its options in the shebang, you might notice that #!/usr/bin/env does not work well with options (depending on OS). Of course, you can write a small wrapper in a separate file. But if you want to do it in one file, you can also do:

#!/bin/bash
# -*- mode: julia -*-
#=
exec julia --color=yes --startup-file=no "${BASH_SOURCE[0]}" "$@"
=#

Base.banner()  # put any Julia code here

Anything between the first #= and exec will be executed as the normal bash command. You can do anything there, like setting up environment variables.

Another tip: In Julia 1.x, you can’t catch InterruptException in the script. However, it is actually capturable in -e ... context. So, as a workaround, you can do:

#!/bin/bash
# -*- mode: julia -*-
#=
exec julia --color=yes --startup-file=no -e "include(popfirst!(ARGS))" "${BASH_SOURCE[0]}" "$@"
=#

@show ARGS
try
    println("sleeping...")
    sleep(10)
catch err
    @show err
finally
    println("woke up")
end

Of course, similar multi-language comment hack works between Python and Julia:

#!/usr/bin/env python
# -*- mode: julia -*-
#=

print("hello from Python!")

import os
cmd = ["julia", __file__]
os.execvp(cmd[0], cmd)

"""
=#

println("hello from Julia!")

#"""
13 Likes
#2

Very neat, thanks! It would be great if the manual had a short section on shebang scripts & Julia, possibly including the above information.

#3

I just sent a PR to include the documentation for SIGINT behavior: https://github.com/JuliaLang/julia/pull/29411

I can send another PR for the “shebang” part. I thought maybe this is too hacky for an official documentation, but let’s see… Not sure where to put, though. https://docs.julialang.org/en/latest/manual/workflow-tips/? https://docs.julialang.org/en/latest/manual/faq/?

1 Like
#4

IMO it should go in Getting started somewhere, since that is the section that talks about running Julia. Could probably be renamed to “Running Julia” too.

1 Like
#5

Here it goes Document scripting tips by tkf · Pull Request #29423 · JuliaLang/julia · GitHub

2 Likes
#6

When I run sbatch xxx.jl file in this way, include function will not find the file path by refering to the xxx.jl file path.
Is this a bug?
Here is the slurm output:

SystemError: opening file "/tmp/slurmd/QuantumOpticsExtra.jl": No such file or directory

The example file tree is this?

DC_1D
├── figdata
│   └── d_loop_data.jl # file sbatched
├── functions.jl # file needed to include
└── QuantumOpticsExtra.jl # file needed to include

example xxx.jl

#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH -J test
#SBATCH --cpus-per-task=9 
#SBATCH -p work
#SBATCH --output=slurm/slurm-%x-%j.out
#SBATCH --error=slurm/slurm-%x-%j.out
#=
exec julia -t$SLURM_CPUS_PER_TASK --project=. --color=yes --startup-file=no "${BASH_SOURCE[0]}" "$@"
=#
using Distributed, SlurmClusterManager
addprocs(SlurmManager(),exeflags=["--project=.", "-t$(ENV["SLURM_CPUS_PER_TASK"])", "--color=yes", "--startup-file=no"])
@everywhere include("../QuantumOpticsExtra.jl")
#7

@Lightup1 I think you’re running into the fact that slurm actually copies your script to a temporary directory when you submit the job. So any files relative to it when you submitted will not be in the same relative place when it runs. I haven’t found an elegant way to deal with this, but this ugly way works. Put something like this immediately above the exec command:

export SLURM_ORIGINAL_COMMAND=$(scontrol show job $SLURM_JOBID | grep "^   Command=" | head -n 1 | cut -d "=" -f 2-)

Then in the julia section, you can figure out the directory like this:

THIS_FILE = if "SLURM_ORIGINAL_COMMAND" ∈ keys(ENV)
    split(lstrip(ENV["SLURM_ORIGINAL_COMMAND"]), " ")[1]
else
    abspath(@__FILE__)
end

The second case is there in case you want to run the script directly, without slurm.