How does one set up a centralized Julia installation?

package
packages
package-manager
installation

#1

I would like to start a thread about best practice for how to set up a centralized Julia installation in a cluster environment or local institution network. Hopefully, this will lead to a tutorial like write-up in the end. I want to focus on Julia 1.0 and, thus, Pkg(3).

How does one set up a system wide Julia 1.0 installation, that is julia itself as well as commonly used packages?

Requirements:

  • Users should still be able to install packages not present in the central directories.
  • As packages might be out of date, users should also be able to install newer versions of packages locally.

From my (very limited) point of view, these could be potential issues:

  • While the system admin updates and precompiles packages of the central installation, running user’s jobs might break because they might try to access the old precompiled resources.

Let me note that this question came up before Julia 1.0 and Pkg3 and has been (sort of) postponed. I thought it might deserve a clean new thread.


System wide Pkg installation
How to create a system administrator depot?
#2

I would also add these few constraints:

  1. how to configure per user where packages will be:
    1.a. downloaded
    1.b. build
    1.c. precompiled
  2. how to configure where custom julia envs will be created:
    2.a. per project
    2.b. per user
  3. are those paths are stacked on top of System Wide ones or they can optionally replace (shadow) each other?
  4. how to merry Julia with external conda environments (to restrict conda to be downloaded by Julia internally and external conda from PATH to be used)
  5. how to merry Julia environments with jupyter notebooks or hub?

To be honest - it would be nice to have a tutorial covering these use cases.


#3

Has there been any update on this? My IT deptartment has very strict home directory quota’s for students which makes it very difficult to teach Julia in the classroom via virtual machines. I have tried to create a project on a shared drive, but that still seems to install packages to local directories.

I have also tried editing the JULIA_DEPOT_PATH to a directory writable by the instructor, but can’t seem to make that work either (referencing: Where or How to edit DEPOT_PATH)


#4

I’d also like to see some helpful feedback here. Currently we also have just a group installation and every package is installed in the users directory, which – as drwx pointed out – is a bit annoying due to quota restrictions.

This also reminds me of a recent discussion about a CLI for installing packages (missing reference) here on discourse.

I’d like to wrap up a workflow in Python which works quite well for us:

  • one centralised Python installation which is occasionally updated (with an official announcement, so everyone is notified and can be prepared)
  • a set of commonly used packages are already installed (Numpy, SciPy and other stuff)
  • users can install packages on their own, using pip install --user PACKAGENAME
  • users can create their own isolated environments to play with or for reproducibility of scientific results etc.

This is more or less done by manipulating $PATH and $PYTHONPATH and at least in my experience it works quite nicely.

What I wish for Julia is indeed something which can be set up using command line tools, so that you can easily provide those setenv.sh scripts for group installations (we use them a lot), or even with the Modules system.

Something like juliapkg install ... to install a package, juliapkg install --user ... to install it in the own JULIA_PKG_DIR or whatever (and found when doing using ... in Julia code), juliaenv create ... and juliaenv activate ... to activate an environment which is then of course used when the user installs new packages via juliapkg install ... etc.

I think this would nice :wink:


#5

I need that as well… Hope someone comes up with a solution soon


#6

I’ve been bashing my head off a wall with this for many hours now. My use case it trying to get Jupyterhub working as a web interface for many users to access Julia. This requires installing the IJulia package in such a way that it is accessible for all users without them first having to SSH in to install it.

I used to be able to do this on v0.6-ish by using the JULIA_PKGDIR environment variable to point to a central directory before installing as root, then adding the same path to the LOAD_PATH julia variable via juliarc.jl – basically what is now referred to as “Old Post” here: https://stackoverflow.com/questions/32338701/install-just-one-package-globally-on-julia

But, this workflow is broken in v1.0 …

I’ve tried replacing JULIA_PKGDIR with JULIA_DEPOT_PATH as suggested in that updated Stack Overflow post, but this creates a lot more than just a package directory in that location (it seems to setup a whole project environment for the root user in the central location). LOAD_PATH then no longer seems to pick that up, so I have to add to the DEPOT_PATH via the startup.jl script and also chmod the directory structure created in that location to be readable by all. These are the steps:

sudo mkdir -p /opt/julia-packages/
sudo sh -c "echo 'push!(DEPOT_PATH, \"/opt/julia-packages/\")' >> /usr/etc/julia/startup.jl"
sudo JULIA_DEPOT_PATH=/opt/julia-packages/ julia -e 'using Pkg; Pkg.add("IJulia")'
sudo chmod -R uog+r /opt/julia-packages

This then works … sort of. It is basically enough to be able to load the Julia 1.0 kernel in Jupyterhub, but if you try to install a package as a user, then contrary to the documentation (which states that all depots except the first are read-only) it tries to install the package in both ~/.julia and in the central depot where IJulia was installed and obviously fails as non-root can’t write there.

Any insights greatly appreciated, because I’m on the point of giving up and removing this feature from my community project.