Worker on cluster : Package not found in current path

General Usage
, ,
1

Hello,
I have been trying to get some packages to run on a cluster. I have set the variable JULIA_DEPOT_PATH to a custom folder where I have installed the required packages and my job script looks as follows:

#==================================================
#PBS -N job_name
#PBS -m abe
#PBS -e <dir_name>
#PBS -o <dir_name>
#PBS -t 0-48
#PBS -l mem=30GB
#PBS -l walltime=2:99:99
#PBS -l nodes=5:ppn=10

export JULIA_DEPOT_PATH=<custom_dir_path>
/dir/julia-1.4.2/bin/julia --machine-file=$PBS_NODEFILE <file_name.jl>

In the file, I have specified using Distributed and @everywhere as required. The problem is as follows: some of the packages load (for ex: JLD and HDF5 load) while other packages do not load (for ex: ITensor, RandomBasedArrays). I have played around with the order of loading packages and hence notice that only few load while the others do not. Also, I have used @everywhere using <package_name> as mentioned in the docs. The error I get is as follows:

ERROR: LoadError: On worker 2:
LoadError: ArgumentError: Package ITensors not found in current path:
- Run `import Pkg; Pkg.add("ITensors")` to install the ITensors package.

The problem seems to be that the packages are not being loaded on the worker. Any pointers on how to fix this would be helpful. Thanks!

2

Are you running a particular environment in file_name.jl? Could you paste the header file of file_name.jl? (i.e. the @everywhere and addprocs commands)?

It might be that you need to pass in the flag --project=/path/to/project to the julia binary

3

The header file for the file_name.jl

using Distributed
@everywhere include("file_1.jl")

and file_1.jl has the following header

using Distributed
@everywhere using Printf, HDF5, ITensors
@everywhere using JLD: save
@everywhere  include("file_2.jl")
@everywhere  include("file_3.jl")

file_2 and file_3 have one dependency each with no @everywhere commands.

Regarding the environment, I am not sure how to set this but previously I was setting JULIA_DEPOT_PATH to a custom package directory (where I have all the packages installed) which worked in v1.0.3.

4

Just a quick update. I got this to work by doing the following in the file file_name.jl:

using Distributed
@everywhere empty!(DEPOT_PATH)
@everywhere push!(DEPOT_PATH, "custom_dir_path")

Thanks for the inputs on this!