How does one set up a centralized Julia installation?

So I just had some time to play around with this. Thanks @fabiangans for your attempt! However, it’s not working properly (and also feels like fighting the package manager :D). I reproduced what you described and observed a couple of issues. Before I get to them, let me add that one should probably export JULIA_PROJECT=/global/depot/path/environments/globalev as administrator as well, as otherwise a v1.0 environment will be created by default which conflicts with same-named user environments.

Some of the issues that I observed with aboves setup:

  • Everytime the admin adds/updates packages he has to remove the General registry folder again. (Maybe you didn’t see this because you said “reverse the order” in which case the users registry will likely be used.)
  • Packages get shared but not their precompilation. Admin can add packages but musn’t precompile them. Otherwise the users julia will try to use the precompiled *.ji files which fails with permission error.

Even worse, there are inconveniences on the user side:

  • If user tries to gc and doesn’t have all the globally installed packages in his current environment (which is basically always the case) he will see warnings:
(v1.0) pkg> gc
    Active manifests at:
        `/home/carsten/.julia/environments/v1.0/Manifest.toml`
┌ Warning: Failed to delete /opt/julia/depot/packages/BenchmarkTools/dtwnm
└ @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:370
   Deleted /opt/julia/depot/packages/BenchmarkTools/dtwnm: 143.768 KiB
┌ Warning: Failed to delete /opt/julia/depot/packages/JSON/Hs3Dj
└ @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:370
   Deleted /opt/julia/depot/packages/JSON/Hs3Dj: 79.503 KiB
   Deleted 2 package installations : 223.271 KiB

And these are just the issues that appeared after 2 minutes of playing around. So this really isn’t a solid setup. Thank you anyway for sharing it with us! It seems to me that the “administrator depot” currently actually isn’t one but is much more “just another depot”.

1 Like

It would be really useful if one of the Pkg developers could comment on either how to do it properly or what the future plan is. Let me (hopefully not rudely) ping @fredrikekre and @StefanKarpinski on this.

Is it/will it be possible to set up a proper administrator depot to provide packages (and maybe their precompiled .ji files) to local users?

(On a personal note, my local system admin is increasingly complaining about all the julia users who all have thousands of redundant files in their home directories.)

2 Likes

Seems like a lot of the issues stems from the fact that we modify all depots. Pkg should probably only modify whats in DEPOT_PATH[1], and https://github.com/JuliaLang/Pkg.jl/pull/733 is one example of that. I guess we should not gc from other depots either, etc etc.

1 Like

As far as I can tell, Pkg trying to update depots that it doesn’t have write access to seems like the only thing preventing this from working currently. The default DEPOT_PATH is something like this:

3-element Array{String,1}:
 "<home>/.julia"
 "<system>/julia/usr/local/share/julia"
 "<system>/julia/usr/share/julia"

The first is the user depot, the second is the shared arch-specific depot and the third is the shared arch-independent depot. There’s not much tooling to help you set this up but it shouldn’t be too hard to do. If you’ve been trying to do this and encountering issues with the basic arrangement then honestly, you’re probably in the best position to help make it work. There are way too many different ways to use the package manager to have actually tried them all out and worked out the rough edges and tooling at this point. We’ve still got our hands full with switching the pkg dev tooling over from METADATA to new registries.

Since Julia published version 1.0, I started exploring Julia today and want to see whether it’s a programming language that can replaces Python. Similar to many others, my first choice is to set it up in JupyterHub so that my learning can be more efficient with notebooks. This issue is really blocking me right now. Hope this can be fixed soon.

The way I currently do things is that I put Julia in /opt/julia and each user sets JULIA_BINDIR appropriately. To be honest, the reason this has always been sufficient is docker: tha majority of work gets done within user directories, anything else gets put in a docker image anyway.

I suppose the way you set up a central installation depends on what you are trying to achieve. A quick and dirty solution might be adding packages that you want in a centralized location using LOAD_PATH (or simply doing Pkg.dev) though this would leave your version control to git itself, so it’s certainly not ideal.

I ran a similar set up mentioned by @fabiangans, but below is what I did in hopefully clearer terms for another user. I still agree with @crstnbr points, but this will work for my use case, for now (until it doesn’t).

It feels to me like PackageSpec should be extendable for an administrator or shared library case, maybe that’s the way to move forward (warning, Julia noob here).


As administrator
Spin up a Julia session and run (this can go into an admins startup.jl):

empty!(DEPOT_PATH)
push!(DEPOT_PATH,"/path/to/shared/packages/") 

Now install a package via the package manager (v1+) then exit, and initialize the package:

(v1.0) pkg> add Gadfly
julia> using Gadfly
julia> exit()

For the User
I have created a startup.jl file that I store for each user in ~/.julia/config/startup.jil . This file looks like:

empty!(DEPOT_PATH)
push!(DEPOT_PATH,string(ENV["HOME"],"/",".julia"),"/path/to/shared/packages/")
push!(LOAD_PATH,"/path/to/shared/packages/")

Then, I just made sure all of the files were rx by all users on the system (not ideal).

2 Likes

Trying out what @drwx suggested.

Minor trouble with the story for developing local packages. Special circumstance here is that users will have restricted internet access. No access to github or curl, ( any command line tools using internet ).

TL;DR how can I prevent adding a shared package to a user developed package from going to github for the registry?

Full story:

Also added a <dev_path> to end of LOAD_PATH for user

I added StaticArrays as administrator. Which is good. Users can “using StaticArrays” just fine.

As user I generated two packages in <dev_path> Foo and Bar.
Foo uses StaticArrays and Bar

from <dev_path> I load julia:

shell> cd Foo
<dev_path>/Foo

(v1.0) pkg> activate .

I added Bar to Foo like so

(Foo) pkg> dev ../Bar
 Resolving package versions...
  Updating `Project.toml`     
  [61032482] + Bar v0.1.0 [`../Bar`]
  Updating `Manifest.toml`          
  [61032482] + Bar v0.1.0 [`../Bar`]

Now I want to add StaticArrays to Foo.

(Foo) pkg> add StaticArrays
  Updating registry at `/path/to/shared/packages//registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: Some registries failed to update:
│     — /path/to/shared/packages/registries/General — failed to fetch 
from repo
└ @ Pkg.API 
/buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:144
 Resolving package versions...
  Updating `Project.toml`
  [90137ffa] + StaticArrays v0.9.2
  Updating `Manifest.toml`
  [90137ffa] + StaticArrays v0.9.2
  [2a0f44e3] + Base64
  ...

(Foo) pkg>

Not a disaster because the Manifest.toml and Project.toml for Foo are good. I can edit Bar and Foo will recompile as a result. So dependencies are working. I can use both Foo and Bar. Yay.

Only real problem here is that the user has to wait ages until the internet check fails when it goes to github or the General Registry. Anything I can do to avoid going to github?

Rereading the above posts and looking at the merged PR, it seems this will be resolved in the next 1.0.x release if I keep my user DEPOT first in that list?

@Orbots I had the same problem.
I have a cluster system that should not connect to the Internet. So first I download my needed packages and past them into .julia/packages/ path. Then to prohibiting the Julia to connect to the Internet, I just simply change the value of repo in the Registry.toml file from https://github.com/JuliaRegistries/General.git to /home/username/.julia/registries/General. It works for me. but how? I don`t know.

@alirezamecheng Nice. That’s a simple solution. Much easier than creating a registry from scratch.
You need to switch repo back if you want to install packages though, eh? You can probably have two Registry.toml files, one for “user” and one for “admin”. But then you need to sync them. Still not ideal, but it would get the job done.

I have a ticket in Pkg.jl project for this. With a potential fix ( I haven’t tested it ). So hopefully this will “just work” soon enough by setting JULIA_DEPOT_PATH and JULIA_LOAD_PATH env variables appropriately for the two roles.


Just wondering if anyone has tested this out with the new Julia 1.1.0 (and possibly a newer build of Pkg)?

My institution is facing the same problem. Yet in our use case, the users do not need to install their own packages, nor to develop their own packages, as the system admin will do it for them.

Hope this can be fixed soon~

I just installed julia 1.1 and tested this out. Happy to report that for this use-case it appears to be fixed, mostly.

You just need to set the JULIA_DEPOT_PATH env var correctly:
This is my setup ( edited from a python script ) where JULIA_CENTRAL_DEPOT holds the path to my central depot.
JULIA_DEPOT_PATH = $HOME/.julia:$JULIA_CENTRAL_DEPOT
JULIA_LOAD_PATH = “@:@v#.#:@stdlib:” + $JULIA_CENTRAL_DEPOT + “:.”

You’ll want to trigger precompilation for all the packages you install as admin or the users will trigger precompilation into the central depot. Unless it is ensured that ~/.julia/compiled/v1.1 path is present ( not present by default ).

Also not great is that if you ]dev a package you are developing it adds it to the environment in the central depot and not ~/.julia/environments which is not something I want.

Anyone know how I can make julia “dev” my local packages into a registryless $HOME/.julia instead of the central depot?

1 Like

Why not use pkg> develop --local Example?

Edit: JULIA_PKG_DEVDIR is probably more appropriate.

1 Like

Don’t want to dictate where a user should develop packages.

Found a couple of fixes here. I was hoping that the default “project” would be the one found at DEPOT_PATH[1]. Which for this setup is $HOME/.julia. Problem is that is the actual default project appears to be the first entry in DEPOT_PATH which has a Project.toml ( and/or Manifest.toml?). If there is no Project.toml in $HOME/.julia it will move on to the next one ( the central depot). By default the $HOME/.julia/logs is the only thing created when running julia for the first time.

The fix: either manually create ~/.julia/environments/v1.1/Project.toml or set JULIA_PROJECT = ~/.julia/environments/v1.1 environment var ( before loading julia ).
JULIA_PROJECT will create the appropriate Project.toml file so next time that var doesn’t have to be set.

Ideally I’d like to set JULIA_DEPOT_PATH = “~/.julia:/path/to/central/depot” and have a brand new user be able to use Julia without causing trouble for others or themselves when they load Julia for the first time. Maybe ~/.julia default directories could be set or an empty Project.toml if needed?

1 Like

JULIA_PROJECT works nicely for creating ~/.julia/environments/v1.1

is there an environment var I can set to force ~/.julia/compiled to be created?

1 Like

@htwai
Just tested with a clean user environment. set JULIA_PROJECT = "~/.julia/environment/v1.1
./julia/compiled is actually created. I might not have tested with the JULIA_PROJECT env var in the previous test.

Everything looks pretty good for the admin/user pattern if you put your central depot path at the end of JULIA_DEPOT_PATH, I have the central depot at the end of my JULIA_LOAD_PATH as well.

Users should set JULIA_PROJECT=~/.julia/environment/v1.1 or load julia with --project=~/.julia/environment/v1.1

For the admin you simply need to switch projects to point into the central repo before adding new packages. $ julia --project=<path_to_central_depot>/environments/v1.1]add` away and users should be able to use them.

3 Likes

Here, with JULIA_PROJECT set, how does a user know which packages are available globally? Running Pkg.installed() returns nothing. Is there a way to specific to look into all depot paths?

You could temporarily switch your project to the central depot.

Pkg.activate(DEPOT_PATH[2]*"/environments/v1.1")

This switches, finds installed packages and then switches back to your original project: Pkg.activate(DEPOT_PATH[2]*"/environments/v1.1"); installed_pkgs = Pkg.installed(); Pkg.activate(DEPOT_PATH[1]*"/environments/v1.1"); installed_pkgs

I see,thank you

See also https://github.com/JuliaLang/Pkg.jl/issues/961