I am currently in the thick of setting up centralized depots on several types of machines (a shared lab macOS system for general use, a couple of shared Linux boxes running instruments, and eventually our cluster accounts). I just spent half the day trying out various scenarios, Because this is being discussed, I decided to drop my notes here.
Overall, I’m impressed with how smoothly it went. Very few issues, and I was able to find workarounds for the hiccups noted below.
My test system was a Mac that just got a fresh install of Julia-1.5.1 straight from the downloaded .dmg package. I tested with a couple of accounts, an “admin” with full permissions and a “user” with restricted permissions. The Julia.app folder is owned by the admin account, as is standard after the install.
I played around with operations as the admin user, with the standard trick of running
popfirst!(DEPOT_PATH) to use the default shared depot location (which happens to be within Julia.app in this scenario, and should work in a similar way on Linux systems). I then tried installing packages from this admin side, and then using them from the user side. The goal is to use the shared depot as much as possible, while allowing the user to install their own (perhaps more recent) copies of code to their local ~/.julia depot.
Wrong default environment for new users
New users will default to using the shared environment, so will try (and fail) to modify the shared
Project.toml. This happens because they will use the first
Project.toml file found in environments in the
DEPOT_PATH, and theirs does not exist yet.
Workaround: First time users should do this (just once):
# yes, twice! Don't try to compose that
(@v1.5) pkg> add Example
(@v1.5) pkg> rm Example
I didn’t try to figure out a cleaner way to make
Project.toml other than this add/rm dance.
pkg> gc as a user fails, with permissions on shared depot
This makes sense, users should not be able to gc the shared depot. But it does block the user’s ability to
Workaround: double pop the
DEPOT_PATH as above, then
Still have to
add foo before
using any packages named
foo that are in the shared depot
This is a bit of an annoyance. But, after adding these packages, Julia will then use the code that sits in the shared depot without creating an unnecessary duplicate in the user depot.
It’s quite possible I’m missing something here (perhaps setting
Packages added to the shared depot should be precompiled at install / update time?
This does not appear to be a big deal. Precompile files go into the user’s
compiled/ dir, which might be the best place for them. For example, if a user overrides a shared package with a newer version, they will be able to precompile the newer version.
Does not seem to be an issue. I tried adding Revise@2.0 to the shared depot (got 2.0.4, all good). Adding Revise to the user environment works, and fetches the correct, most-recent version (2.7.something, IIRC). As does precompilation.
Install as an admin as usual into shared depot. Then, from within the same julia session, run
notebook() as described in IJulia’s readme. Will prompt for the miniconda install; accept.
Ran into an issue as a user. After adding IJulia to the user environment,
using IJula errors:
julia> using IJulia
ERROR: InitError: could not load library "/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib"
dlopen(/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib, 1): no suitable image found. Did find:
/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib: stat() failed with errno=13
chmoding all of the artifacts in that dir to 755 allowed me to run IJulia as a user, without generating a whole new copy of miniconda (in fact, no ~/.julia/conda folder at all. Woot!).
notebook() seemed to run, although I didn’t set up a proper tunnel to that machine to test the web interface.
This permissions issue looks like something that could be fixed in Pkg.
Our lab code is hosted in a custom registry. These seem to just work. I was able to add the custom registry (as a firstname.lastname@example.org url), and did not run into errors updating as the normal user. However, note that users will not be able to update the custom registry on their own, this must be done by the admin user working in the shared depot. This is not a big problem, however, as these registries are not changing as much as General. I suppose that users could also add the custom registry on their own to allow for faster updating, but I did not test that workflow.
When using email@example.com urls, be sure to start the ssh-agent and ssh-add when working remotely. Look into https://www.funtoo.org/Keychain which was recommended here https://github.com/JuliaLang/Pkg.jl/issues/911#issuecomment-640908492