Julia Slurm Cluster Module Command Not Working

General Usage
, ,
5

Thanks for the follow-up information! Despite my unfocused blanket questions :sweat_smile: I think we found the issue.

I just checked on our cluster (we also use Slurm + Lmod) and get the same behavior. The issue seems to be what you already mentioned in the original post:

I found similar descriptions here and on StackOverflow explaining the problem and the corresponding part of the docs:

  • The command is never run with a shell. Instead, Julia parses the command syntax directly, appropriately interpolating variables and splitting on words as the shell would, respecting shell quoting syntax. The command is run as julia’s immediate child process, using fork and exec calls.

Short summary:

The shell, e.g. bash or zsh, is a program (executable file/script somewhere on the system) that usually runs when you open a terminal and is used to run all kinds of other programs. Running commands in the shell will either look for another executable file in the system’s path variable(s) and run that or if the command is “built-into” the shell (or defined as a function in some shell script), run that. module seems to be the latter.

Since Julia doesn’t run commands in a shell if we use run(cmd), only actual executables in the path will be found. This explains also why run(`/usr/bin/bash -c module list`) worked (we first run the shell and then have the module shell function available again. Similarly, using the shell mode with ; will use a shell as well, so that also works.

Here’s a way to tell what is built-in and what is not (the which command looks for the executable file that corresponds of the command)
EDIT: That’s not quite correct, please check the next reply by @sijo for an important clarification!

  • which ls will print something like /usr/bin/ls
  • which which → /usr/bin/which
  • but which source will give which: no source in (..long list of paths to look into..), the source command loads shell code from a file
  • and similarly which module also fails, which indicates that module is indeed not a executable file somewhere on the system, but only works within a shell

This is an issue with command parsing. As far as I understand, since () can be used in interpolation, e.g. like this:

julia> a = "hello"
"hello"

julia> cmd = `echo $(a)`
`echo hello`

we have to “qoute” the parentheses if they should appear in the final command.

But module("list") from the link you mentioned looks like a call to a Python function and it wouldn’t work in a shell most likely. The syntax for invoking a command with arguments is usually using spaces instead of parentheses, as in echo hello (not echo(hello)).

1 Like