Just wondering if anyone has tested this out with the new Julia 1.1.0 (and possibly a newer build of Pkg)?
My institution is facing the same problem. Yet in our use case, the users do not need to install their own packages, nor to develop their own packages, as the system admin will do it for them.
Hope this can be fixed soon~
I just installed julia 1.1 and tested this out. Happy to report that for this use-case it appears to be fixed, mostly.
You just need to set the JULIA_DEPOT_PATH env var correctly:
This is my setup ( edited from a python script ) where JULIA_CENTRAL_DEPOT holds the path to my central depot.
JULIA_DEPOT_PATH = $HOME/.julia:$JULIA_CENTRAL_DEPOT
JULIA_LOAD_PATH = â@:@v#.#:@stdlib:â + $JULIA_CENTRAL_DEPOT + â:.â
Youâll want to trigger precompilation for all the packages you install as admin or the users will trigger precompilation into the central depot. Unless it is ensured that ~/.julia/compiled/v1.1 path is present ( not present by default ).
Also not great is that if you ]dev a package you are developing it adds it to the environment in the central depot and not ~/.julia/environments which is not something I want.
Anyone know how I can make julia âdevâ my local packages into a registryless $HOME/.julia instead of the central depot?
Why not use pkg> develop --local Example?
Edit: JULIA_PKG_DEVDIR is probably more appropriate.
Donât want to dictate where a user should develop packages.
Found a couple of fixes here. I was hoping that the default âprojectâ would be the one found at DEPOT_PATH[1]. Which for this setup is $HOME/.julia. Problem is that is the actual default project appears to be the first entry in DEPOT_PATH which has a Project.toml ( and/or Manifest.toml?). If there is no Project.toml in $HOME/.julia it will move on to the next one ( the central depot). By default the $HOME/.julia/logs is the only thing created when running julia for the first time.
The fix: either manually create ~/.julia/environments/v1.1/Project.toml or set JULIA_PROJECT = ~/.julia/environments/v1.1 environment var ( before loading julia ).
JULIA_PROJECT will create the appropriate Project.toml file so next time that var doesnât have to be set.
Ideally Iâd like to set JULIA_DEPOT_PATH = â~/.julia:/path/to/central/depotâ and have a brand new user be able to use Julia without causing trouble for others or themselves when they load Julia for the first time. Maybe ~/.julia default directories could be set or an empty Project.toml if needed?
JULIA_PROJECT works nicely for creating ~/.julia/environments/v1.1
is there an environment var I can set to force ~/.julia/compiled to be created?
@htwai
Just tested with a clean user environment. set JULIA_PROJECT = "~/.julia/environment/v1.1
./julia/compiled is actually created. I might not have tested with the JULIA_PROJECT env var in the previous test.
Everything looks pretty good for the admin/user pattern if you put your central depot path at the end of JULIA_DEPOT_PATH, I have the central depot at the end of my JULIA_LOAD_PATH as well.
Users should set JULIA_PROJECT=~/.julia/environment/v1.1 or load julia with --project=~/.julia/environment/v1.1
For the admin you simply need to switch projects to point into the central repo before adding new packages. $ julia --project=<path_to_central_depot>/environments/v1.1 ]add` away and users should be able to use them.
Here, with JULIA_PROJECT set, how does a user know which packages are available globally? Running Pkg.installed() returns nothing. Is there a way to specific to look into all depot paths?
You could temporarily switch your project to the central depot.
Pkg.activate(DEPOT_PATH[2]*"/environments/v1.1")
This switches, finds installed packages and then switches back to your original project: Pkg.activate(DEPOT_PATH[2]*"/environments/v1.1"); installed_pkgs = Pkg.installed(); Pkg.activate(DEPOT_PATH[1]*"/environments/v1.1"); installed_pkgs
I see,thank you
Here is what I contributed to the julia-in-production BoF channel at JuliaCon 2020:
Hi all,
I work as a computational scientist at the Swiss National Supercomputing Centre (CSCS) and am the responsible for Julia computing at CSCS. We have put Julia in September last year in production on the Piz Daint GPU supercomputer which hosts 5704 GPUs [0] (2017/2018 Piz Daint was during 1 year listed as number 3 of the world on top500.org). In August, we will also make it available on the CSCS jupyterhub service. BTW, we have also a page on the CSCS user portal dedicated to Julia [1].
I will try to summarize a little bit our approach in providing Julia on the supercomputer and point out difficulties and things that might could be improved on the Julia side, and in particular on the package manager side.
At CSCS, all the scientific software stack is built with the Python-based âEasyBuildâ [2]. It is important to know that Piz Daint contains a GPU partition (âgpuâ) and an multicore partition (âmcâ). So we created an EasyBuild recipe for two modules âJuliaâ and âJuliaExtensionsâ for each partition, âgpuâ and âmcâ (as it is done for all the scientific software stack). So the modules contain the following:
The modules set up a stacked environment. The easyconfig files [3-6] are very high-level and declarative, the magic happens in the easyblock files [7-9]. The easyblock files are quite difficult to read if one is not used to working with these files. I will try to summarize the most important. E.g. for the Julia-gpu module, it sets the following environment variable:
setenv JULIA_CUDA_USE_BINARYBUILDER false
setenv JULIA_PROJECT ~/.julia/1.4.2/daint-gpu/environments/1.4.2-daint-gpu
setenv JULIA_DEPOT_PATH ~/.julia/1.4.2/daint-gpu:<path-CUDA-MPI>:<path-Julia>
setenv EBJULIA_USER_DEPOT_PATH ~/.julia/1.4.2/daint-gpu
setenv EBJULIA_ADMIN_DEPOT_PATH <path-CUDA-MPI>:<path-Julia>
setenv JULIA_LOAD_PATH @:@#.#.#-daint-gpu:<path-CUDA-MPI>:<path-Julia>/environments/1.4.2-daint-gpu:@stdlib
setenv EBJULIA_USER_LOAD_PATH @:@#.#.#-daint-gpu
setenv EBJULIA_ADMIN_LOAD_PATH <path-CUDA-MPI>/environments/1.4.2-daint-gpu:@stdlib
where we have:
<path-CUDA-MPI>: path to CUDA.jl and MPI.jl
<path-Julia>: path to Julia base installation
Then, then EasyBuild installation also creates a Julia âstartup.jlâ file which fixes the DEPOT_PATH and the LOAD_PATH to get (there is a little bit more coding in there):
DEPOT_PATH .= [USER_DEPOT_PATH; ADMIN_DEPOT_PATH]
LOAD_PATH .= [USER_LOAD_PATH; ADMIN_LOAD_PATH]
The result for the user is the following:
when loading Julia[Extensions], Julia and the additional packages are immediately usable; precompiled files go into the users home, e.g. ~/.julia/1.4.2/daint-gpu for GPU.
when a user installs a package it goes by default in his home, e.g. ~/.julia/1.4.2/daint-gpu for GPU.
a user can install a different version of the provided package and it will automatically take his installation as the USER_DEPOT_PATH and the USER_LOAD_PATH have precedence.
Now, letâs talk about what could maybe be improved from my point of view. They main things that come to my mind are:
It is not very fortunate to need to do a reshuffling of the DEPOT_PATH and the LOAD_PATH in startup.jl. This breaks for example the unit tests from certain packages, in particular when called directly from the package manager. It would be good if the Pkg manager would allow to set ADMIN_DEPOT_PATH, USER_DEPOT_PATH, ADMIN_LOAD_PATH and USER_LOAD_PATH and create from this the DEPOT_PATH and LOAD_PATH.
Quering the available packages in the stacked environment is not obvious. Pkg manager could improve this.
Then, this works all pretty well for small or medium jobs, but for large jobs with thousands of GPUS it is not good that each process needs to read from the same files (stacked environment / home). For large-scale deployment, it would be nice if one could create a binary / folder that can be copied at the beginning of a job to the compute nodes ram-disk (/tmp) and then each compute node just reads from there during the job. I have successfully run with up to thousand GPUs using such an approach in a âmanual wayâ. It would be great if e.g. PackageCompiler.jl could help with that. BTW: a discussion that I initiated on this topic on Julia discourse a year ago did not lead to a solution then [10].
We have the intention to share our EasyBuild recipies on the official EasyBuild repository. To this, we are however still missing some work: building Julia from source, making parts of it maybe more generic and do some cleanup.
@Fredrik Ekre and @Kristoffer Carlsson, could you comment on the points related to the Pkg manager and PackageCompiler.jl?
Looking forward to a great and productive discussion with all of you!
Thanks!!
Sam
REFERENCES
I agree. We should not assume that /home/user/.julia is the place where Julia keeps software.
As @samo points out this does not scale on big HPC systems - I have worked with systems where /home is deliberatley quotaed very small.
But why do you not then use JULIA_PROJECT and JULIA_DEPOT_PATH which are located on a fast filessystem? Clearly I am about to be schooled hereâŚ
First of all, we are still far from a perfect solution. Besides, like you can see looking at the never-ending discussions in Julia discourse, it is not at all obvious how an ideal solution should look like.
We have opted for installing the most important packages for HPC as MPI.jl and CUDA.jl into an ADMIN_DEPOT_PATH, which is on a different file system than home. However, the user must be able to install his own packages and for this the most practical default is currently the home (the user can of course change this to scratch for example, but files on scratch are wiped out after 30 days without usageâŚ).
Finally, note that the current solution is fine for small and medium jobs, but far from ideal for large jobs. See point 3 above.
I hope this reply might revive this topic again as we are all still lacking an ideal solution.
Thanks Sam, that was a good overview.
What happens when a new release of e.g. MPI.jl is tagged: do you need to manually go through and add that to the shared depot? Otherwise wouldnât it install it in the userâs depot?
Sorry for the late reply. I somehow missed it.
We install MPI.jl and CUDA.jl together with the Julia base module. Of course, the user can install a newer version of MPI.jl and CUDA.jl in his user space if he needs it; it will take precedence in the stacked environment.
Our usual scientific software stack is only updated a few times a year as most users use the supercomputer for production rather than development. So, they typically want to have the same software version during the whole allocation of a project.
I am currently in the thick of setting up centralized depots on several types of machines (a shared lab macOS system for general use, a couple of shared Linux boxes running instruments, and eventually our cluster accounts). I just spent half the day trying out various scenarios, Because this is being discussed, I decided to drop my notes here.
Overall, Iâm impressed with how smoothly it went. Very few issues, and I was able to find workarounds for the hiccups noted below.
My test system was a Mac that just got a fresh install of Julia-1.5.1 straight from the downloaded .dmg package. I tested with a couple of accounts, an âadminâ with full permissions and a âuserâ with restricted permissions. The Julia.app folder is owned by the admin account, as is standard after the install.
I played around with operations as the admin user, with the standard trick of running popfirst!(DEPOT_PATH) to use the default shared depot location (which happens to be within Julia.app in this scenario, and should work in a similar way on Linux systems). I then tried installing packages from this admin side, and then using them from the user side. The goal is to use the shared depot as much as possible, while allowing the user to install their own (perhaps more recent) copies of code to their local ~/.julia depot.
New users will default to using the shared environment, so will try (and fail) to modify the shared Project.toml. This happens because they will use the first Project.toml file found in environments in the DEPOT_PATH, and theirs does not exist yet.
Workaround: First time users should do this (just once):
pop!(DEPOT_PATH)
pop!(DEPOT_PATH)
# yes, twice! Don't try to compose that
(@v1.5) pkg> add Example
(@v1.5) pkg> rm Example
I didnât try to figure out a cleaner way to make Project.toml other than this add/rm dance.
pkg> gc as a user fails, with permissions on shared depotThis makes sense, users should not be able to gc the shared depot. But it does block the userâs ability to gc.
Workaround: double pop the DEPOT_PATH as above, then gc.
add foo before using any packages named foo that are in the shared depotThis is a bit of an annoyance. But, after adding these packages, Julia will then use the code that sits in the shared depot without creating an unnecessary duplicate in the user depot.
Itâs quite possible Iâm missing something here (perhaps setting LOAD_PATH).
This does not appear to be a big deal. Precompile files go into the userâs compiled/ dir, which might be the best place for them. For example, if a user overrides a shared package with a newer version, they will be able to precompile the newer version.
Does not seem to be an issue. I tried adding Revise@2.0 to the shared depot (got 2.0.4, all good). Adding Revise to the user environment works, and fetches the correct, most-recent version (2.7.something, IIRC). As does precompilation.
Install as an admin as usual into shared depot. Then, from within the same julia session, run notebook() as described in IJuliaâs readme. Will prompt for the miniconda install; accept.
Ran into an issue as a user. After adding IJulia to the user environment, using IJula errors:
julia> using IJulia
ERROR: InitError: could not load library "/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib"
dlopen(/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib, 1): no suitable image found. Did find:
/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib: stat() failed with errno=13
Sure enough, chmoding all of the artifacts in that dir to 755 allowed me to run IJulia as a user, without generating a whole new copy of miniconda (in fact, no ~/.julia/conda folder at all. Woot!). notebook() seemed to run, although I didnât set up a proper tunnel to that machine to test the web interface.
This permissions issue looks like something that could be fixed in Pkg.
Our lab code is hosted in a custom registry. These seem to just work. I was able to add the custom registry (as a git@github.com url), and did not run into errors updating as the normal user. However, note that users will not be able to update the custom registry on their own, this must be done by the admin user working in the shared depot. This is not a big problem, however, as these registries are not changing as much as General. I suppose that users could also add the custom registry on their own to allow for faster updating, but I did not test that workflow.
When using git@github.com urls, be sure to start the ssh-agent and ssh-add when working remotely. Look into Funtoo Keychain Project - Funtoo which was recommended here https://github.com/JuliaLang/Pkg.jl/issues/911#issuecomment-640908492
I would love to know whether, 2 years later, a tutorial or a recommend procedure has been set up for a Linux box.
This might be beneficial also to inform the installation of jupyterhub on a server.
also trying to setup a Jupyter server right now by deciphering the discussion above âŚ
I am years late, but this has a renewed interest since we want to âforce installâ in a centralized way packages in v1.9 [weakdeps]. Itâs the same reasons for many on this thread that compute nodes donât have Internet access and multiple users package duplication. On a related topic, is there an option in Pkg.instantiate() to force install [weakdeps] ?
Probably here you get better, more up to date, tips: Installing packages via an SSH SOCKS proxy on a compute cluster