How to install packages on an offline cluster?

New to Julia
1

I’m trying to use julia on a cluster, which does not have usual network,
so I have to install the packages offlinely.

I have read several post regarding this problem,
it seems that I should upload the entire ~/.julia folder onto the cluster.

However, since I am using Mac and the cluster is CentOS on amd,
this method cannot be used.

Is there any way to solve this problem?

2

Did you try it? I would assume that this works, because macOS and CentOS are UNIX-based (but this is just a guess).
Alternatively you could set up a virtual machine with CentOS, install everything as desired, and copy the ~/.julia folder from the VM to your cluster.

3

Unfortuantely, I have tried, and it won’t work.

I am installing virtual system now, but I highly doubt whether it will work,
since the hardwares are so different.

4

I think there are 2 things, depending on what you want to do:

  1. Install a single package to an existing installation:
    copy the downloaded git over and do the usual “]add foldername” in the same way as you would do “]add url”
  2. Deploy a Julia installation with existing packages installed:
    I use the following powershell script to create a relocatable installation with preinstalled packages (needs recompiling though). In the folder that contains this script, you need to put a juliainstall folder that contains the main Julia files that you can just download from the website.
$FILEDIR=Get-Location
# Set env variables required to avoid loading stuff from user directory
$env:JULIA_DEPOT_PATH=Join-Path $FILEDIR "jlresources"
$env:JULIA_LOAD_PATH = $env:JULIA_DEPOT_PATH + ";" + $FILEDIR
$env:JULIA_PROJECT = $FILEDIR

$JULIAPATH = Join-Path $FILEDIR "\juliainstall\bin\julia.exe"

# Start julia with forwarded arguments
& $JULIAPATH $args
5

I was not aware that the content in ~/.julia is hardware dependent. Another approach could be:

  • When you start a julia instance on the cluster the first time, it should automatically create a new ~/.julia folder there.
  • copy the desired packages in the ~/.julia/packages/ folder from local your machine to the respective folder on the cluster. I would not expect that data to be hardware dependent.
  • Perhaps also copy the ~/.julia/environments/v1.10/Manifest.toml from your local machine to the cluster (assuming use julia 1.10).
8

If the packages have any binary dependencies, there will definitely be architecture-specific files in .julia/artifacts.

10

Wasn’t the whole point of this question that the target environment couldn’t download stuff?

1 Like
11

I realize now my previous answer was not very helpful. Your approach with the VM would also be my best bet, unless you have an Mac with arm chip. Did it work that way?
You will just have to “]precompile” on the new system first though.