Package management - an intro?

General Usage
1

I see that there is a talk by Stefan Karpinski on Pkg3 package management.
Python ‘virtualenv’ is mentioned. Would anyone be kind enough to point me towards a gentle intro to Julia package management.
My specific interest is managing Julia installaitons and package on HPC clusters, where there is traditionally a central installation on a shared network drive. However people can and do want to install specific packages or bleeding edge versions for their own use.

2

Package management is one of the more complicated julia topics and if you are interested in the future, read https://github.com/JuliaLang/Juleps/blob/master/Pkg3.md and see Stefan’s talk and then wait until Pkg3 shows up, because currently it’s not visible.

On the current package management tools handling of a central place for packages and mix it with local packages looks like an issue. You can read about https://docs.julialang.org/en/stable/manual/packages/ in the manual. The current approach is to have a package installation per user and it might be possible to link the directory to another, central place but then you have to deal with user rights issues and similar. Search here or in the old Redirecting to Google Groups forum.

1 Like
3

Andreas, thankyou for the reply.
The ‘depots’ feature sounds ideal for the environments I am talking about.

The ‘Environments’ section looks great too.
HPC setups normally use software modules Environment Modules (software) - Wikipedia
It should be easy to set up a Modules file to load a particular TOML file.
Things are looking more sunny on the package selection side!

4

Actually, my understanding is that it is totally possible to do this and have it co-exist with user-installed packages in the current version of Julia.

As sysadmin, suppose you want to install into a shared directory /foo/julia. Create /foo/julia/v0.5 for the packages (assuming you are using Julia 0.5) and create /foo/julia/lib/v0.5 for the precompiled cache files. Then, in Julia, do:

v = string('v',VERSION.major,'.',VERSION.minor) # v0.5 or similar
ENV["JULIA_PKGDIR"] = joinpath("/foo/julia", v)
Base.LOAD_CACHE_PATH[1] =  joinpath("/foo/julia/lib", v)

and install and compile each package Foo that you want to be installed system-wide:

Pkg.add("Foo")
import Foo

(You may need a chmod -R on /foo/julia to make sure it has the desired permissions.)
Finally, in each user’s .juliarc file, add:

let v = string('v',VERSION.major,'.',VERSION.minor)
    push!(joinpath("/foo/julia",v), LOAD_PATH)
    push!(joinpath("/foo/julia/lib",v), Base.LOAD_CACHE_PATH)
end

and the system-wide packages will co-exist with user-installed packages. When a user does using Foo, it will first look in the user’s package directory, and then look in the system-wide directory if the user didn’t install their own Foo.

5

So, do you run a setup like this or do you think a setup like this will run?

6

I don’t do it myself, but I think it will work; that is certainly the intention of how the package loading is designed. And my impression from github issues (e.g. Running julia from multiple machines sharing the same home folder · Issue #13684 · JuliaLang/julia · GitHub and Pkg directory shared among users has several permission problems · Issue #12876 · JuliaLang/julia · GitHub) is that various people have gotten variations on this idea to work reasonably well, although it could be made smoother.

7

We run a system now where packages are stored in /usr/local/julia/julia-0.6/usr/share/julia/site for a julia installation at /usr/local/julia/julia-0.6/. Precompiled libs are stored in individual user’s .julia directories, but the source is used from the central store unless the user has her/his own checkout of the package. That provides a fair amount of flexibility. The main downside occurs when the admin (me) forgets to update the packages sufficiently frequently, and this causes users to do more and more of their own package management.