Sharing Julia Depots on Multi-User Platforms

General Usage
1

On a multi-user Linux platform, you want to install a Julia environment once while it is being used by many users (possibly at the same time). One can create a central installation of a Julia environment by exploiting the JULIA_DEPOT_PATH environment variable, which therefore also provides a means to have multiple different Julia environments. Our problem is that Julia requires this JULIA_DEPOT_PATH to be a writeable location, in particular to create log files in a subdirectory logs, which are dependent on the user that uses the environment. However, on a multi-user platform, you want a separation between a centrally installed application and anything that is user or usage specific.

Therefore, we would like to specify a different location for logging files (and anything else that is user or usage specific). Setting JULIA_HISTORY to ~/.julia/logs didn’t work and we couldn’t find a way to make the location of the Julia environment at JULIA_DEPOT_PATH independent of what a specific user does. Perhaps the approach with JULIA_DEPOT_PATH isn’t the right approach. Any suggestions of how to fix this?

We are working with Julia 1.10.0

1 Like
2

For my lab server each user has its own environment.
Instead, I install some basic packages automatically when the user is created.

3

Have you considered having multiple depots? Only the first depot of the stack is supposed to be writable, all the others can safely be read-only. So for each user you could have

export JULIA_DEPOT_PATH=":/path/to/shared/depot"

and no one would clash with anyone else. There would still be some duplication across users, but a lot less than the no-shared-depot scenario.

8 Likes
4

I got confused for a bit because environments usually refer to something else in Julia, I’m going to say depots here. You reasonably want users to have separate REPL histories and other background log files, but can you further clarify what you want to be shared versus separate? For example, is the shared depot intended for users to share installed code (versions of packages, artifacts, etc)? Do you want users to have an option to store installed code in that central depot, or just their own?

5

I have tried the following. I created a symbolic link for $JULIA_DEPOT_PATH/logs to /tmp, which is a writeable location. This works as long as mutltiple users don’t access the scratch_usage.toml file at the same time (Not sure if it is also a problem when a single user accesses this file with multiple runs as the same time). So, even if the location would be writable, the whole idea doesn’t work in a multi-user context. Any ideas on an alternative solution as not supporting this renders Julia to be no option at all :frowning:

6

As sharing depot already share code. Setting up LocalRegistry.jl and share installation scripts is my current way.

7

Yes, I suggested it above. Sharing files between different users in Unix is just a pain, whatever tool you use, it’s not really specific to julia. My suggestion is not to do it. Hence why I recommend using a shared non-writable depot, but it must not be the first one of the stack.

1 Like
8

The solution you describe is similar to what I did by creating this symbolic link to a writable location (see above). Unfortunately, it does not work when multiple users access the same depot (at the top-level) at the same time as you get file write access conflicts resulting in Julia to crash…

Edit: an extra observation is that the generate file permissions are wrong to allow write access by multiple different users, which basically disables having a shared depot:

-rw-rw-r-- 1 user user 13082 Jun 18 13:51 scratch_usage.toml

In my opinion, having the generated file permissions in the Julia source code fixed wouldn’t be a good solution. I would suggest to have the Julia source code use a temporary file for this :wink:

9

What we are looking for the basic principle of Separation of Concerns. Any Tool related stuff should be separated from user/usage related stuff. In our case, we do not want individual users to change anything in the shared installed code (Tool). However, I can imagine the idea of users having the ability to extend it with packages for their own good. In such case, the extensions for that user should be at a different location, i.e., a location where only that user has access to.

To answer concretely:

Yes, in terms of using that shared code. But not in terms of changing that shared code.

Only their own

Julia is our first case of tooling for which we encounter this problem. We don’t have problems with for example centrally installed Python environments or tools like Matlab at non-writable locations.

10

Something similar to what giordano suggested and the linked documentation explains should do that, but your use of the word “environment” is giving me pause again, especially since you use in the context of Python where the word does mean something similar to environments in Julia. When you say Python environment, do you mean virtual environments? If so, how exactly are you letting users share a read-only environment but install their own libraries on top of it?

11

Ok, I didn’t read this well enough! I missed the “:” :yawning_face: This does actually do what we want! THANK YOU!

2 Likes