@Orbots I had the same problem.
I have a cluster system that should not connect to the Internet. So first I download my needed packages and past them into
.julia/packages/ path. Then to prohibiting the Julia to connect to the Internet, I just simply change the value of
repo in the
Registry.toml file from
/home/username/.julia/registries/General. It works for me. but how? I don`t know.
@Orbots I had the same problem.
@alirezamecheng Nice. That’s a simple solution. Much easier than creating a registry from scratch.
You need to switch
repo back if you want to install packages though, eh? You can probably have two Registry.toml files, one for “user” and one for “admin”. But then you need to sync them. Still not ideal, but it would get the job done.
I have a ticket in Pkg.jl project for this. With a potential fix ( I haven’t tested it ). So hopefully this will “just work” soon enough by setting
JULIA_LOAD_PATH env variables appropriately for the two roles.
Just wondering if anyone has tested this out with the new Julia 1.1.0 (and possibly a newer build of Pkg)?
My institution is facing the same problem. Yet in our use case, the users do not need to install their own packages, nor to develop their own packages, as the system admin will do it for them.
Hope this can be fixed soon~
I just installed julia 1.1 and tested this out. Happy to report that for this use-case it appears to be fixed, mostly.
You just need to set the JULIA_DEPOT_PATH env var correctly:
This is my setup ( edited from a python script ) where JULIA_CENTRAL_DEPOT holds the path to my central depot.
JULIA_DEPOT_PATH = $HOME/.julia:$JULIA_CENTRAL_DEPOT
JULIA_LOAD_PATH = “@:@v#.#:@stdlib:” + $JULIA_CENTRAL_DEPOT + “:.”
You’ll want to trigger precompilation for all the packages you install as admin or the users will trigger precompilation into the central depot. Unless it is ensured that ~/.julia/compiled/v1.1 path is present ( not present by default ).
Also not great is that if you
]dev a package you are developing it adds it to the environment in the central depot and not ~/.julia/environments which is not something I want.
Anyone know how I can make julia “dev” my local packages into a registryless $HOME/.julia instead of the central depot?
Why not use
pkg> develop --local Example?
JULIA_PKG_DEVDIR is probably more appropriate.
Don’t want to dictate where a user should develop packages.
Found a couple of fixes here. I was hoping that the default “project” would be the one found at DEPOT_PATH. Which for this setup is $HOME/.julia. Problem is that is the actual default project appears to be the first entry in DEPOT_PATH which has a Project.toml ( and/or Manifest.toml?). If there is no Project.toml in $HOME/.julia it will move on to the next one ( the central depot). By default the $HOME/.julia/logs is the only thing created when running julia for the first time.
The fix: either manually create ~/.julia/environments/v1.1/Project.toml or set JULIA_PROJECT = ~/.julia/environments/v1.1 environment var ( before loading julia ).
JULIA_PROJECT will create the appropriate Project.toml file so next time that var doesn’t have to be set.
Ideally I’d like to set JULIA_DEPOT_PATH = “~/.julia:/path/to/central/depot” and have a brand new user be able to use Julia without causing trouble for others or themselves when they load Julia for the first time. Maybe ~/.julia default directories could be set or an empty Project.toml if needed?
JULIA_PROJECT works nicely for creating ~/.julia/environments/v1.1
is there an environment var I can set to force ~/.julia/compiled to be created?
Just tested with a clean user environment. set JULIA_PROJECT = "~/.julia/environment/v1.1
./julia/compiled is actually created. I might not have tested with the JULIA_PROJECT env var in the previous test.
Everything looks pretty good for the admin/user pattern if you put your central depot path at the end of JULIA_DEPOT_PATH, I have the central depot at the end of my JULIA_LOAD_PATH as well.
Users should set JULIA_PROJECT=~/.julia/environment/v1.1 or load julia with --project=~/.julia/environment/v1.1
For the admin you simply need to switch projects to point into the central repo before adding new packages.
$ julia --project=<path_to_central_depot>/environments/v1.1 ]add` away and users should be able to use them.
Here, with JULIA_PROJECT set, how does a user know which packages are available globally? Running
Pkg.installed() returns nothing. Is there a way to specific to look into all depot paths?
You could temporarily switch your project to the central depot.
This switches, finds installed packages and then switches back to your original project:
Pkg.activate(DEPOT_PATH*"/environments/v1.1"); installed_pkgs = Pkg.installed(); Pkg.activate(DEPOT_PATH*"/environments/v1.1"); installed_pkgs
I see,thank you
Here is what I contributed to the julia-in-production BoF channel at JuliaCon 2020:
I work as a computational scientist at the Swiss National Supercomputing Centre (CSCS) and am the responsible for Julia computing at CSCS. We have put Julia in September last year in production on the Piz Daint GPU supercomputer which hosts 5704 GPUs  (2017/2018 Piz Daint was during 1 year listed as number 3 of the world on top500.org). In August, we will also make it available on the CSCS jupyterhub service. BTW, we have also a page on the CSCS user portal dedicated to Julia .
I will try to summarize a little bit our approach in providing Julia on the supercomputer and point out difficulties and things that might could be improved on the Julia side, and in particular on the package manager side.
At CSCS, all the scientific software stack is built with the Python-based ‘EasyBuild’ . It is important to know that Piz Daint contains a GPU partition (‘gpu’) and an multicore partition (‘mc’). So we created an EasyBuild recipe for two modules ‘Julia’ and ‘JuliaExtensions’ for each partition, ‘gpu’ and ‘mc’ (as it is done for all the scientific software stack). So the modules contain the following:
- Julia-mc : Julia + MPI.jl
- Julia-gpu : Julia + MPI.jl + CUDA.jl
- JuliaExtensions [5,6]: some additional packages like Plots.jl etc.
The modules set up a stacked environment. The easyconfig files [3-6] are very high-level and declarative, the magic happens in the easyblock files [7-9]. The easyblock files are quite difficult to read if one is not used to working with these files. I will try to summarize the most important. E.g. for the Julia-gpu module, it sets the following environment variable:
setenv JULIA_CUDA_USE_BINARYBUILDER false setenv JULIA_PROJECT ~/.julia/1.4.2/daint-gpu/environments/1.4.2-daint-gpu setenv JULIA_DEPOT_PATH ~/.julia/1.4.2/daint-gpu:<path-CUDA-MPI>:<path-Julia> setenv EBJULIA_USER_DEPOT_PATH ~/.julia/1.4.2/daint-gpu setenv EBJULIA_ADMIN_DEPOT_PATH <path-CUDA-MPI>:<path-Julia> setenv JULIA_LOAD_PATH @:@#.#.#-daint-gpu:<path-CUDA-MPI>:<path-Julia>/environments/1.4.2-daint-gpu:@stdlib setenv EBJULIA_USER_LOAD_PATH @:@#.#.#-daint-gpu setenv EBJULIA_ADMIN_LOAD_PATH <path-CUDA-MPI>/environments/1.4.2-daint-gpu:@stdlib
where we have:
<path-CUDA-MPI>: path to CUDA.jl and MPI.jl <path-Julia>: path to Julia base installation
Then, then EasyBuild installation also creates a Julia ‘startup.jl’ file which fixes the DEPOT_PATH and the LOAD_PATH to get (there is a little bit more coding in there):
DEPOT_PATH .= [USER_DEPOT_PATH; ADMIN_DEPOT_PATH] LOAD_PATH .= [USER_LOAD_PATH; ADMIN_LOAD_PATH]
The result for the user is the following:
when loading Julia[Extensions], Julia and the additional packages are immediately usable; precompiled files go into the users home, e.g. ~/.julia/1.4.2/daint-gpu for GPU.
when a user installs a package it goes by default in his home, e.g. ~/.julia/1.4.2/daint-gpu for GPU.
a user can install a different version of the provided package and it will automatically take his installation as the USER_DEPOT_PATH and the USER_LOAD_PATH have precedence.
Now, let’s talk about what could maybe be improved from my point of view. They main things that come to my mind are:
It is not very fortunate to need to do a reshuffling of the DEPOT_PATH and the LOAD_PATH in startup.jl. This breaks for example the unit tests from certain packages, in particular when called directly from the package manager. It would be good if the Pkg manager would allow to set ADMIN_DEPOT_PATH, USER_DEPOT_PATH, ADMIN_LOAD_PATH and USER_LOAD_PATH and create from this the DEPOT_PATH and LOAD_PATH.
Quering the available packages in the stacked environment is not obvious. Pkg manager could improve this.
Then, this works all pretty well for small or medium jobs, but for large jobs with thousands of GPUS it is not good that each process needs to read from the same files (stacked environment / home). For large-scale deployment, it would be nice if one could create a binary / folder that can be copied at the beginning of a job to the compute nodes ram-disk (/tmp) and then each compute node just reads from there during the job. I have successfully run with up to thousand GPUs using such an approach in a “manual way”. It would be great if e.g. PackageCompiler.jl could help with that. BTW: a discussion that I initiated on this topic on Julia discourse a year ago did not lead to a solution then .
We have the intention to share our EasyBuild recipies on the official EasyBuild repository. To this, we are however still missing some work: building Julia from source, making parts of it maybe more generic and do some cleanup.
@Fredrik Ekre and @Kristoffer Carlsson, could you comment on the points related to the Pkg manager and PackageCompiler.jl?
Looking forward to a great and productive discussion with all of you!
 Piz Daint | CSCS
 Run a julia application at large scale (on thousands of nodes) - #6 by johnh
I agree. We should not assume that /home/user/.julia is the place where Julia keeps software.
As @samo points out this does not scale on big HPC systems - I have worked with systems where /home is deliberatley quotaed very small.
But why do you not then use JULIA_PROJECT and JULIA_DEPOT_PATH which are located on a fast filessystem? Clearly I am about to be schooled here…
First of all, we are still far from a perfect solution. Besides, like you can see looking at the never-ending discussions in Julia discourse, it is not at all obvious how an ideal solution should look like.
We have opted for installing the most important packages for HPC as MPI.jl and CUDA.jl into an ADMIN_DEPOT_PATH, which is on a different file system than home. However, the user must be able to install his own packages and for this the most practical default is currently the home (the user can of course change this to scratch for example, but files on scratch are wiped out after 30 days without usage…).
Finally, note that the current solution is fine for small and medium jobs, but far from ideal for large jobs. See point 3 above.
I hope this reply might revive this topic again as we are all still lacking an ideal solution.
Thanks Sam, that was a good overview.
What happens when a new release of e.g. MPI.jl is tagged: do you need to manually go through and add that to the shared depot? Otherwise wouldn’t it install it in the user’s depot?
Sorry for the late reply. I somehow missed it.
We install MPI.jl and CUDA.jl together with the Julia base module. Of course, the user can install a newer version of MPI.jl and CUDA.jl in his user space if he needs it; it will take precedence in the stacked environment.
Our usual scientific software stack is only updated a few times a year as most users use the supercomputer for production rather than development. So, they typically want to have the same software version during the whole allocation of a project.
I am currently in the thick of setting up centralized depots on several types of machines (a shared lab macOS system for general use, a couple of shared Linux boxes running instruments, and eventually our cluster accounts). I just spent half the day trying out various scenarios, Because this is being discussed, I decided to drop my notes here.
Overall, I’m impressed with how smoothly it went. Very few issues, and I was able to find workarounds for the hiccups noted below.
My test system was a Mac that just got a fresh install of Julia-1.5.1 straight from the downloaded .dmg package. I tested with a couple of accounts, an “admin” with full permissions and a “user” with restricted permissions. The Julia.app folder is owned by the admin account, as is standard after the install.
I played around with operations as the admin user, with the standard trick of running
popfirst!(DEPOT_PATH) to use the default shared depot location (which happens to be within Julia.app in this scenario, and should work in a similar way on Linux systems). I then tried installing packages from this admin side, and then using them from the user side. The goal is to use the shared depot as much as possible, while allowing the user to install their own (perhaps more recent) copies of code to their local ~/.julia depot.
Wrong default environment for new users
New users will default to using the shared environment, so will try (and fail) to modify the shared
Project.toml. This happens because they will use the first
Project.toml file found in environments in the
DEPOT_PATH, and theirs does not exist yet.
Workaround: First time users should do this (just once):
pop!(DEPOT_PATH) pop!(DEPOT_PATH) # yes, twice! Don't try to compose that (@v1.5) pkg> add Example (@v1.5) pkg> rm Example
I didn’t try to figure out a cleaner way to make
Project.toml other than this add/rm dance.
pkg> gc as a user fails, with permissions on shared depot
This makes sense, users should not be able to gc the shared depot. But it does block the user’s ability to
Workaround: double pop the
DEPOT_PATH as above, then
Still have to
add foo before
using any packages named
foo that are in the shared depot
This is a bit of an annoyance. But, after adding these packages, Julia will then use the code that sits in the shared depot without creating an unnecessary duplicate in the user depot.
It’s quite possible I’m missing something here (perhaps setting
Packages added to the shared depot should be precompiled at install / update time?
This does not appear to be a big deal. Precompile files go into the user’s
compiled/ dir, which might be the best place for them. For example, if a user overrides a shared package with a newer version, they will be able to precompile the newer version.
Does not seem to be an issue. I tried adding Revise@2.0 to the shared depot (got 2.0.4, all good). Adding Revise to the user environment works, and fetches the correct, most-recent version (2.7.something, IIRC). As does precompilation.
Install as an admin as usual into shared depot. Then, from within the same julia session, run
notebook() as described in IJulia’s readme. Will prompt for the miniconda install; accept.
Ran into an issue as a user. After adding IJulia to the user environment,
using IJula errors:
julia> using IJulia ERROR: InitError: could not load library "/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib" dlopen(/Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib, 1): no suitable image found. Did find: /Applications/Julia.app/Contents/Resources/julia/local/share/julia/artifacts/58a71f17a58227868e74248b4a07c5b7d277b1cd/lib/libzmq.5.dylib: stat() failed with errno=13
chmoding all of the artifacts in that dir to 755 allowed me to run IJulia as a user, without generating a whole new copy of miniconda (in fact, no ~/.julia/conda folder at all. Woot!).
notebook() seemed to run, although I didn’t set up a proper tunnel to that machine to test the web interface.
This permissions issue looks like something that could be fixed in Pkg.
Our lab code is hosted in a custom registry. These seem to just work. I was able to add the custom registry (as a firstname.lastname@example.org url), and did not run into errors updating as the normal user. However, note that users will not be able to update the custom registry on their own, this must be done by the admin user working in the shared depot. This is not a big problem, however, as these registries are not changing as much as General. I suppose that users could also add the custom registry on their own to allow for faster updating, but I did not test that workflow.
When using email@example.com urls, be sure to start the ssh-agent and ssh-add when working remotely. Look into Funtoo Keychain Project - Funtoo which was recommended here https://github.com/JuliaLang/Pkg.jl/issues/911#issuecomment-640908492
I would love to know whether, 2 years later, a tutorial or a recommend procedure has been set up for a Linux box.
This might be beneficial also to inform the installation of jupyterhub on a server.
also trying to setup a Jupyter server right now by deciphering the discussion above …