I’ll illustrate a little vignette of how global environment bloat becomes painful for new users:
Make a few simple plots based on a tutorial using a handful of packages installed in the global environment
As the project’s scope expands, gradually add packages
Perform a throwaway analysis for a separate project that requires one or two more packages (also added to the global environment)
A third project appears. This time, two of the necessary packages are incompatible, but a fix is available on #master
This breaks something deep inside the graphics stack for the first project
[two hours of package resolution hell and refactoring just because the user wanted to make a plot]
All this pain would be avoided, of course, if users created a new environment for each project, but as multiple commenters have mentioned, the necessity of siloing projects is not drilled into new users who are happy to take the path of least resistance. The problem isn’t the existence of a persistent global environment, but rather that the global environment is the default. It shields new users from the complexity of Pkg, projects, manifests, environments, etc., but that debt can only be deferred for so long before it needs to be collected (with interest).
Speaking personally, I regularly do quick and ad-hoc testing in my default global environment, adding packages willy-nilly. It’s quick, convenient and easy. But then I’ll also remove them, either when I’m done or at some other point down the road. And I’ll take the time to throw things into a project environments the moment it goes from REPL play into a .jl file. If I’m touching the filesystem then I have a directory for it and it means I might come back to it.
I have begun to do most of my simple checks (either for MWE for the forum or for my own play) only after ] activate --temp. But this not really a great solution in my opinion.
A solution might be to have a flag like --constrained. Using it in package installation would force to look for the latest version of PackageToBeInstalled that does not implies an update on any other package of the current environment.
This would not change the separation of generating the Manifest.toml and loading the package, and the version resolution would occur in package installation. I guess this still have the following problem.
(Bold is mine.)
It is also a really hard problem to solve and implies a lot of work. I guess similarly as @StefanKarpinski says here (I know this is for loading, but the work I guess is similar for installing).
For the following I think that making a GUI-like tool would help a lot to solve the problem, it is a graph dependency visualization utility, so should not be that hard to make.
I just noticed that when one installs a package that depends on another package, but that other package is loaded in a incompatible version from the shared environment, we get this very gentle warning:
1 dependency precompiled but a different version is currently loaded.
Restart julia to access the new version.
So it seems that that problem is already present in the current workflow. Does not seem different to me, and a warning / error message may suffice.
I normally use temporary environments for all my try-and-trash code, I even have a little script to open Julia in that mode (with OhMyREPL and TerminalPager).
This is it
julia -q --startup-file=no -e 'using OhMyREPL, TerminalPager; ENV["PAGER"] = "julia_pager.sh"; using Pkg; Pkg.activate(temp=true)' -i
where julia_pager.sh is
#!/usr/bin/env bash
bat -l julia -n --color=always "$2" | less -R $1
Yes, that is exactly right. I’m not claiming it would solve all problems, but I think it is a fairly simple (and hopefully not too controversial or disruptive) change that would really, really help a ton. I think it would essentially just align the actual behavior with what most users think the implementation is (or at least the vast majority of new to medium experienced users). So my vote would be that we try that (or something similar) for say 1.9, and then we can still have a long and extensive discussion for a perfect solution, but in the meantime we have something good
It is true that when a package is auto-installed into an environment, it cannot change the versions of packages that are already loaded. However, the issue is limited to only new packages you try to load into the environment. That is relatively uncommon since the packages you commonly use are already in your default environment with versions that have already been resolved to be compatible.
The difference is that when you restart Julia it starts with a correctly resolved set of package versions including the one you just added and got this warning for, so when you do the same sequence of imports you don’t get an error and have compatible versions of all the packages. In your proposal on the other hand, the environment is ephemeral, so when you restart Julia and do the same sequence of imports, you get into the exact same broken state with the same warning because there is no memory of wanting that set of packages. Permanence of the environment is exactly the feature that allows for not repeating the same mistake over and over again.
Note that there can be situations where even though there is a compatible set of versions of the packages you want to use there is no possible order of package imports that leads to a compatible set of versions if you commit to the best version of each package before considering the other packages. You need to have a global view of the set of packages that you want to use when picking versions in order to be able to find all possible solutions. Here’s an example:
you want to use packages A and B
A has versions [2, 1]
B has versions [2, 1]
C has versions [3, 2, 1]
A and B both depend on C
A (all versions) is incompatible with C3
B (all versions) is incompatible with C2
Here’s the conundrum:
If you load A first, you get A2 and C2 which is incompatible with B;
If you load B first, you get B2 and C3 which is incompatible with A.
If you can reason about it globally, however, you can see that A1, B1, C1 is a compatible set of versions, it’s just not one you can find with a greedy algorithm.
I appreciate the time you take to explain those things, and I really feel uncomfortable following the discussion (given that surely you have better things to do). But the changes in “my proposal” to adapt to that could be very simple (and maybe even could be the default behavior): just throw an error if an incompatibility is found, suggesting perhaps the user to update things.
Actually this could be the behavior now, and not letting the user in a possibly broken Julia session (I’m not claiming it should be, but it seems that could well have been an acceptable decision with other tradeoffs).
The biggest issue with doing this is that there is no syntax in JULIA_LOAD_PATH for expressing that you want @v#.# in the load path only if the active project entry @ doesn’t resolve to anything. The default value is @:@v#.#:@stdlib. What you’re suggesting would be making the default something like this instead: @|@v#.#:@v#.#-global:@stdlib, with the assumption that we make | special syntax for “the first of these that exists”.
Yes, that was my idea: always activate v1.7 (unless of course a project is specified via the JULIA_PROJECT env var or command line flags), and just do entirely away with the situation where there is no active project. And then the JULIA_LOAD_PATH would be @:@v#.#-global:@stdlib.
That might even simplify things in the code base more generally because one could now rely on there always being an active project, i.e. no longer would one need to special case handle the situation where there is no active project.
First of all, I’m glad to see that there is a chance to have this changed this time!
I’m wondering what could be a better name for what is currently called v#.#-global here. IMO, the global is misleading because the regular v#.# environment is also global in some sense (irrespective of where one starts Julia, it get’s automatically loaded if no explicit active project is specified),
Would it make sense to consider introducing a command line option that simply disables the stacking in the sense that it drops @v#.#-global from the default LOAD_PATH stack when asked for?
Yeah I didn’t want to bike shed the name here so I just went with David’s suggestion to explore the concept. Stacking can already be disabled by setting JULIA_LOAD_PATH=@. Who knows if Windows has mktemp? Not me.
Yep, and I’m not really sure whether “global” is a good name for that either, I just picked something
I think if we were starting from scratch, it might make sense to call all projects in .julia/environments “global” and the one that is always loaded “shared”, but of course that is no longer an option, because the term “shared” is already used for everything in the .julia/environments folder, so ideally we would find a good new name for the one that would always be loaded
I experience this almost weekly. What’s unfortunate is that most of the time I am loading the same packages (Plots, CUDA etc), and the latency involved feels unnecessary.
Thanks for the proposed workflow, I hope that it speeds up this process.
)