Julia on remote RedHat cluster

New to Julia
1

Dear all,

in order to conduct numerical calculations I have been granted access to a remote high-performance cluster running on Redhat linux. So far, I can run a batch script that calls a simple julia file with:

println(ENV[“PATH”]);
println(Base.load_path());

which returns:

/apps/compilers/julia/1.3.1/tools:/apps/compilers/julia/1.3.1/bin:/apps/compilers/java/9.0/jdk-9.0.4/bin:/apps/compilers/gnu/4.9.2/bin:/apps/compilers/intel/18.0.4/vtune_amplifier/bin64:/apps/compilers/intel/18.0.4/debugger_2018/gdb/intel64_mic/bin:/apps/compilers/intel/18.0.4/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/usr/lpp/mmfs/bin:/users/pa19/lanast/bin:/users/pa19/lanast/bin

[“/users/pa19/lanast/.julia/environments/v1.3/Project.toml”, “/users/apps/compilers/julia/1.3.1/share/julia/stdlib/v1.3”]

The admins advised that certain variables must be set, in order to be able to fetch and run all the Julia packages:
#!/bin/bash
JULIAROOT=$HOME/julia
export PATH=$JULIAROOT/bin:$JULIAROOT/tools;
export LD_LIBRARY_PATH=$JULIAROOT/lib:$JULIAROOT/lib64;
export INCLUDE=$JULIAROOT/include;

Now, do I have to type in these commands in the terminal? Or do I have to place them in side an .sh or an .jl file? And do I need to include the “export” command?

Thank you, all the best
Lysandros

2

Customizations and how jobs can be run should be documented for your cluster, it is not something that is specific to Julia.

Perhaps talk to the sysadmins, or a user who is experienced using this cluster and has a setup to share.

3

I agree with @Tamas_Papp talk to your friendly sysadmins. Promis to bring them cookies or beer when the current fuss is over.

Many HPC clusters use the ‘modules’ environment to set these variables.
type ‘module avail’ to see if there is a Julia module. Also ask the sysadmins about modules.

I would put those environment variables at the start of your job script if there is no module available.

4

Something else to ask your cluster admins.
Which filesystem should you use for storing data and programs.
Normally the /home directory is on a smaller, low performance filesystem.
There probably will be a large capacity high performance filesystem.

5

Thank you @Tamas_Papp and @johnh for the advice.

Strangely enough, the second element in the bracket of the vector which the command println(Base.load_path()); returns, is a folder on the server which contains all the familiar packages like “LinearAlgebra” and “Statistics”. But the packages still do no not load when I try simple commands.

Could someone guide me to documentation for the difference between Load, Load_path, Juliaroot and LD_library_path.
This would be helpful.

Best wishes

P.S. julia/1.3.1 is included in the list of module avail

6

After doing some some research, it seems, that one has to append to the Load Path environmental variable, the directory where all the packages are installed in the user directory of the cluster, which is good.
Now I only have to do the same with the Project.toml file…Is there a way to tell Julia to use a different one, than the one listed in Base.load_path() ?