How to create a system administrator depot?

[carstenbauer]

This is from the Pkg section of the Julia doc about the DEPOT_PATH:

The first entry is the “user depot” and should be writable by and owned by the current user. The user depot is where: registries are cloned, new package versions are installed, named environments are created and updated, package repos are cloned, newly compiled package image files are saved, log files are written, development packages are checked out by default, and global configuration data is saved. Later entries in the depot path are treated as read-only and are appropriate for registries, packages, etc. installed and managed by system administrators.

How do I set up such a system administrator depot?

Basically, what I want is to install a couple of packages system wide such that users don’t have to install them by themself.

(This is in the context of How does one set up a centralized Julia installation?)

[carstenbauer]

If possible, I would not just like to provide raw packages (stuff in packages/) but also the precompiled .ji files (stuff in compiled/).

A user, when he first starts julia, and types add XY will just get a reference to this system wide installation of XY in his Projects.toml. When he then tries to using XY it should just work (without another precompilation).

UPDATE: Ok, I probably don’t need the compiled/ stuff to be system wide.

[jlperla]

@carstenbauer

UPDATE: Ok, I probably don’t want the compiled/ stuff to be system wide.

Why not? The versions are segmented effectively? I am trying to get something similar built and curious on your thinking.

[carstenbauer]

@jlperla, I guess these files are rather small and there are few of them. Hence it’s not too importat to provide them in a centralized fashion. Also (guessing here) they might depend on the user’s system (image). Is that right?

(Concretely, I am looking at Conda, which ends up really big when no system python is used. Turns out that it’s the dependencies in packages/Conda/deps which (understandably) blow up.)

[jlperla]

I see. My main concern was caching the precompiled version to speed up the user experience.

[johnh]

@carstenbauer Can you point to a specific problem here?
One thing to note - the Modules environment is always useful in HPC clusters
https://modules.sourceforge.net
I have simple Modules file which I use to work on either 0.6.4 and 1.0.0
I think we should recognize that there will ALWAYS be a need for old/current/test versions of software packages on a shared system.
When Julia 1.1 debuts it might be marvellous. But no-one will move wholesale to ti without being confident that 1.0.0 is available. Modules makes this easy.

[carstenbauer]

@johnh, I’m well aware of modules. In fact, I kind of recommended them two days ago (Handling multiple versions of Julia - #2 by carstenbauer)

I want to set up a centralized installation of julia in a multi-user institution network. So far (<v0.7) I could basically only provide the binaries and every user has it’s own .julia. This leads to packages and so on being installed many many times.

WIth Pkg3 in Julia 1.0 it should be possible to also have some popular packages in a centralized julia-depot folder somewhere (see doc quote above).

[johnh]

@carstenbauer Thankyou. That makes a great deal of sense. If you want to bounce some ideas off me, feel free.

[CDLuminate]

Hi @carstenbauer, have you managed to setup such a depot?

I looked into how the slug is calculated and managed to install packages to the /usr/share/julia directory (the last path in DEPOT_PATH). But the problem is I still have to manually add the metadata into ~/.julia/environments/v1.0/Manifests.toml and Project.toml in order import it from julia. For example

⛬ >using Compat
⛬ >pathof(Compat)
"/usr/share/julia/packages/Compat/6B9KA/src/Compat.jl"

Do you have idea about this?