I will add one additional feature which, that one, would actually guide the users to the use of environments, in a nicer way: saving the current temporary environment and the history to a file. Such as:
julia> using Draft
Activating new project at `/tmp/jl_AUMyBc`
julia> @reuse StaticArrays Plots
julia> plot(Tuple.(rand(SVector{2,Float64}) for _ in 1:100))
julia> savefig("test.pdf")
julia> save_project("./my_project")
Now, this save command would create:
the Project.toml file in the my_project directory.
the Manifest.toml file there.
my_project/MyProject.jl, containing the history of the REPL:
using StaticArrays
using Plots
plot(Tuple.(rand(SVector{2,Float64}) for _ in 1:100))
savefig("test.pdf")
(the better this file reflected the last state of the REPL, the better, although that could be hard, but anyway the user can then edit that file and move forward, now, in a reproducible state, created from this exploratory REPL code, if he/she decided that to be interesting).
Saving the current REPL session would for sure be a nice feature.
I think only IJulia has this option, but I could be wrong.
The big repl_history.jl file is also not that useful for this, I fear.
Is there a good way to track the commands entered in a specific REPL session?
Maybe a pre/post execution hook to also reset the recording when the environment is changed?
Yes, I very strongly agree with that! I have observed literally dozens and dozens of Julia users here in an academic setting going from beginner and pretty knowledgeable users, and this trips up almost all of them. I mean, not 70%, I mean more like 99% And I am only exaggerating a very, very tiny bit here.
I think there is also a very easy and simple fix for this: have one shared project (in the Pkg.jl sense, i.e. simply a project that lives in .julia/environments) named v1.7global that is always on in the stack of available env (i.e. essentially what currently is v1.7). And then make a change that there is always a project active in any Julia session, and if the user doesnât specify anything activate the shared env v1.7. Done, I think that would resolve a huge amount of confusion.
The main benefit is that novice users would normally add stuff to the v1.7 project, and if they then activate a different project, that stuff would no longer hang around. If someone wants to make a package always available, then they need to go through extra hoops, which would be much better because that is the much less common scenario. For now they could just activate the v1.7 project, add a dep, and be done, but one could also use syntax like ]add MyPackage --project=@v1.7shared or something like that.
This does not solve the dependency nightmare of the cluttered shared environment. Putting users in a temporary environment solves that and this other confusion.
Ah. Well that explains why Julia doesnât already launch with julia --project=@. option by default!.
A small tweak/security enhancement
What if a message popped-up for every new directory-based-environment that was run? Something like:
You have requested to start a new Julia session using the environment at /path/to/current/working/directory
This does not correspond to any directory on your list of trusted Julia environments.
How would you like to proceed?
â [T]rust: Add to my list of trusted environments, and proceed to the Julia REPL
â [P]roceed to the Julia REPL using this environment only for this session.
â [A]bort (default).
List of trusted environments
Could possibly be stored in a file like ~/.julia/trusted
Possibly requiring some form of password encryption to add to the list of trusted paths?
(Hoping this encryption doesnât become easily decipherable given the likelyhood of having similar repeating patterns⌠Iâm no expert on encryption.)
Sorry. I donât understand. The way you explain it sounds exactly like how .julia/environments/v1.7 already works⌠Just that you want there to be a second one called v1.7global for some reason.
I definitely understand observing a large percentage of Julia users navigating environments with an ad-hoc approach. I have experienced that first-hand myself (with my own semi-blind trial-and-error approach). I simply I donât understand how this particular proposal helps. I must be getting snagged somewhere in the explanation.
agree, but why have a shared global one all to fulfill this? Vscode would handle it seamlessly already if you always have a project file available when opening with Julia --project. Want to share project files?Put in common folder, or at least have a common parent. Come back to run some code six months later and no problems at all. Only stack development tools.
Exactly. The issue is there is no way to have a shared project file with a non-trivial number of dependencies and not have it constantly break conda-style.
It does not. But itâs a rather simple change with a lot of positive effect. I think switching to your proposed temporary environment will be more complicated to implement and also expect it to face more social resistance.
(Personally I donât mind a cluttered global environment so much. Itâs rather simple to understand. And if it becomes a problem you can learn about temporary environments or Draft.jl or whatever other nice things we come up with. The conflation of a global default and global shared environment is much more unfortunate and unexpected IMO.)
The difference is that currently v1.7 is always available, even if you activate some other project. With my proposal, v1.7 would no longer be active if you activate some other project.
Yes, I totally agree. There is absolutely nothing surprising about that part and it is also something that is not bothering me at all. But this business of stacked environments, and that by default most users add stuff to a project that is always active just is something that almost all newish users donât understand and is causing a lot of grief. In particular because it really torpedos the workflow of having a separate project in a local folder. I canât even tell you how many times someone sends me a folder with a project and some code, and they think everything is self-contained, but then it turns out it isnât because they are using some package that is in their v1.7 project, but not in their local project, and they never notice that because things of course run just fine on their machine.
A different solution to this would be to make loading from the load path disallowed when running a script in an active project. That introduces a deviation between the REPL and the script, though, which might not be ideal. Deactivating @v#.# when thereâs an active project is an interesting approach. It doesnât entirely disallow making a script thatâs not self-contained since what youâre calling @v#.#-global would still be accessible from a script. Youâre really relying on the assumption that only people who know what theyâre doing would put anything in @v#.#-global at all and that they would presumably avoid depending on things in there from their scripts. If you really want to enforce that scripts in active projects are standalone, then you canât allow even that.
I understand where youâre coming from here, but Iâll try to explain why this proposed approach is not a good one.
When you say âthe most recent version available (without fetching any info from the web!)â youâre glossing over a lot of complexity. Picking a compatible, working set of version is one of the hardest and most important things that a package manager does for you and in this scheme it cannot do that. Taken at face value your proposal means that we chose literally the most recent version that is already installed. Which means completely ignoring compatibility with packages that are already loaded. This means that people would constantly end up with broken combinations of package versions that are known not to work together. That would be a very bad user experience.
Ok, so what if we try to avoid that? We could try to pick the most recent version of Package that is not incompatible with any package versions that weâve already loaded. That might work a bit better, but it amounts to doing package version resolutionâa notoriously hard problemâon the fly with a greedy algorithm that cannot backtrack, since you cannot unload an already loaded package version. Thatâs very likely to frequently back users into corners where they want to load some package but there is no version thatâs possible.
Thereâs also the issue that itâs not great to put so much complex logic on the critical path of package loading. Julia already does quite a lot here by parsing the manifest and using the contents of the manifest to find the right versions of packages to load. But it was carefully designed to be fairly minimal and straightforward and not depend too much on the contents of the rest of your hard drive. The steps to load a package are:
Parse the active manifest
Find the stanza for the current module
Look up the UUID of Package in that stanza
Find the stanza for Packageâs UUID
Look up the version in that stanza
Find the fixed version path in a depot.
By step five you have the name, UUID and content-hash (or explicit path for devâd packages), so you have the exact relative path inside of a depot that the right package version will be found at and you just have to look in each depot at that known location. The total work required to load a package is parsing one TOML file, doing some reasoning based on its content, and then looking for a fixed path in each depot (and itâs usually in the first one).
In what youâre proposing, on the other hand, the loading code would have to look through every depot for every installed version of Package and parse the project TOML file in each one to extract a version number. It would then sort the version numbers and pick the latest one. If we want to not load known incompatible versions of packages, itâs worse: then we need to load all registries and parse the Compat TOML files for Packageand for every package that weâve loaded so far to make sure that they are compatible. Thatâs a completely over the top amount of complex work to do at package load time and it depends on all sorts of state all over your system.
One of the hard design constraints that Pkg had to satisfy was that package loading has to be fairly simple. This is in conflict with the fact that picking a compatible set of package versions is very complex. The solution Pkg uses is to separate manifest generation from package loading: manifest generation is done by Pkg and can be as complex as it needs to be; once a manifest is generated, however, loading packages based on it is straightforward. Picking package versions at load time, as youâre proposing, throws that clean separation out the window and forces us to solve a really complex problem (version resolution) at a time when we really donât want to be solving that kind of problem (during package loading).
Stepping back, I think that the objection to a persistent global environment is quite poorly motivated. The entire objection appears to be that it can get âbloatedâ. But why does that matter? Why is bloat a problem? Are you concerned about disk space? Compatibility problems? Whatâs the concern? Presumably your concern isnât disk space since your proposal doesnât address disk space at all as it doesnât lead to fewer package versions being installed. So the issue much be compatibility issues? But as pointed out above, itâs far worse at dealing with compatibility problems than what we currently have. If the latest versions of all packages you want to use happen to be compatible with each other than your proposal does work nicely; but in that case thereâs also no problem with having them all in a global environment. So can you give clearer motivation for why having a persistent global environment is a problem?
[Note: I made some substantial edits a few minutes after posting to improve clarity]
How is this different from what may happen when loading a package available in the @v1.7 shared environment when using another environment? Shouldnât the possible issues be exactly the same? Sincere question, here, is there any difference? My current impression is that the package available on the shared environment is just loaded for good.
(reason which avoiding this is something suggested there as well, and cause of other usability issues that David is more concerned about).
Well, my experience is that:
After a while, if there are too many packages there, installing or updating anything starts to become very slow. Some packages are simply impossible to keep available in a shared environment (CUDA, in particular), because having there implies that I may get an update of that at some time and have to wait several minutes to work on what I wanted.
By default, installing anything there may trigger a cascade of updates of everything. Also very inconvenient if the purpose is to do some quick exploratory code.
From time to time, one of these packages fail to keep the Compat entry up to date, and then it holds back the updating of other packages, finally leading to a compatibility roadblock. Finding which package is holding other back is not that straightforward.
These points make of the current default environment a inadequate place to write exploratory code, follow package tutorials, execute example scripts, etc. Thus, my current âexploratoryâ developing mode involves 1) starting a temporary environment; 2) using Pkg.offline() 3) add-ing the packages I need there; 4) using those packages and, finally, doing what I need to do.
That has worked very well, except for the steps needed, instead of just using the packages installed in the @v1.7 and coding, which was what I used to do before facing the problems with the bloated shared environment.
So, instead of having new Julia users (not hard core developers) facing those problems, it seemed to me that a default temporary environment could be something more safe and error free. With that they could follow tutorials, test examples, etc, without worrying about having any long-term consequences on their Julia installation.
Because my workflow of writing exploratory code in a temporary and offline environment has worked mostly issue free, I thought that having that as a more standard option (of course pushing it to be the standard is way out of my scope) could be nice.
In my dreams, having a
julia --draft
startup with more or less those properties (and with the possibility of saving the history and environment at some point!) would be a great usability feature.
But no more than that, I donât think that Julia will vanish without that
And I am only exaggerating a very, very tiny bit here.
This does not correspond to any directory on your list of trusted Julia environments.