Hello,
I want to setup Julia with (of course) some packages on a HPC which does not have access to internet. This means I can only SCP from my local machine to the HPC.
In this case, how can I manually install packages for julia?
Consider that Julia is loaded as a module, and I do not have root access.
Thank you
If the environment allows some kind of container (eg a Docker image), then that’s the ideal solution.
Otherwise, a poor man’s replacement would be generating a ~/.julia directory in a clean environment, compressing and copying it, for which you can just write a simple script using Pkg.instantiate etc.
I would echo the container solution. For example, a Singularity image for interactive sessions. For job composing, if your HPC is using Slurm, you should be able to activate the correct environment and have it working will with the Julia module. For installing other packages it might be tricky since some packages need to build the modules against that system… I would install first the package and download all the artifacts… then I would try and test at what point you should copy the files and build in the HPC or locally.
JuliaTeam also offers support in helping getting the infrastructure set up in HPC and secure environments.
Might be enough to install everything on the login node and then set JULIA_DEPOT to point to the .julia in the home folder of the login node. It depends a bit on how the file system works etc.
Thank you for your answers. I am sorry but I do not know how to create a Docker image, nor I know how to do any of your suggesions !
Could you please be more explicit, or maybe give me some resources about this topic?
I don’t think my system administrators will do anything for me though. It took a lot of persuasion to make them install Julia already…
Can’t I just extract the right packages for the HPC in my .julia folder?
The biggest problem is going to be binary dependencies: this might work if you’re on the same OS, but probably not otherwise.
Yes, you should be able to stage the packages in your ~/.julia/ folder. But as @simonbyrne cautioned, make sure your workstation runs the same architecture/ABI as your HPC, but tryig any project that uses binary dependencies should establish whether that is a problem.
I would use rsync -av ~/.julia/ dest:~/.julia/ where dest is the destination host, which will make a complete copy the first time and then make incremental updates on subsequent invocations, compressing everything in transit and using ssh to make the connections. The trailing / are significant.
Even if you believe that your sysadmins would not help you with the details, they could point you in the right direction. This is a problem that all users depending on nontrivial setups encounter, so there may be a recommended way to do it on your cluster, but we don’t have the information to help you with that.
Docker images are well documented, just search the web. It takes about an afternoon to set up one from scratch for the first time. But perhaps you should defer learning about them until you confirm that it is possible to use one.
Most HPC clusters I have used in an academic setting have a variety of support channels, from the tutorials to user forums. I would recommend investing in learning about these, the problem is not Julia-specific.