Oh, another point I’d like to make: The temporary depot created is 0.7GB, which is a lot of complex software that was downloaded, installed and worked flawlessly on many different systems, including multiple different graphics stacks. The experience in so many programming languages trying to get this kind of stuff installed and working is having it take days to get right, if ever. We’re not even talking about if things install anymore—it’s taken as a given! Anyway, a couple of minutes to install a fully working graphics stack is a delight.
31 Likes
I think the issue is that since Pluto is reactive, there’s an expectation that everything is supposed to be snappy. But installing and loading packages from the internet just isn’t. So the solution seems to me to be to separate the UX for things that are known to be slow and not reactive from the things that are part of the real reactive experience. Get package installation out of the reactive UX!
19 Likes
I think this is a pretty clever idea! Have an import form where the user enters packages they want, and there could even be a submenu that shows versions available with an indicator or highlighting for versions which are already installed.
6 Likes
Another phrasing is “one-off and ad hoc analysis becomes 85% slower, while running the same script repeatedly becomes 33% faster”.
If that’s the choice, I personally wouldn’t switch to “new pkgimages” in day-to-day use – only for very specific workloads. Lots of work I do in Julia is exploratory analysis, that will only be run a few times. These scenarios hardly gain anything from new pkgimages, and would work better with the pre-1.9 approach.
Best of both worlds for interactive work would be to run “old” precompilation at first, and present a ready to use REPL/notebook immediately after that. Then, run the “new” native code precompilation in background when no user code executes – and save native pkgimages after it completes. Does this sound feasible?
4 Likes
I don’t think that’s an accurate phrasing at all. It’s only installation of packages that got slower, which is not what “one-off and ad-hoc analysis” suggests. You can do one-off ad hoc analysis all day with a fixed set of packages installed. In my experience I install new packages pretty infrequently—I have a set of tools that I use and needing a new one isn’t all that common once they usual suspects are available.
9 Likes
Unfortunately though (stop me if I sound like a broken record) for this to be true in Pluto, it’s going to need to be smarter about reusing package versions used in previous notebooks
The problem is basically just that with Plutos automagic package installation and one-off environments, if you have
- one notebook that uses packages A, B, and C which you made on monday
- another using A, B, and D which you made on wednesday
- and a third one using B, D, and C which you made on friday
then there’s a very real possibility that you end up needing precompile each of packages A, B, C, and D two separate times. This isn’t something that’s likely to happen if you just use the global environment, but with Pluto it definitely does happen.
4 Likes
Pluto is just one instance of this, but the issue is more general.
Do you mean using a single environment for everything? It goes against common recommendations (that I totally agree with) of using separate envs for each analysis/project.
3 Likes
One workflow that might be nice interactively is something like ] activate --copy which creates a new environment that copies the current Project.toml and Manifest.toml over. That way you don’t have to pay a precompilation price when doing one-off things, but your state is saved if you want to return to it (even if your main environment changes).
5 Likes
If I’m doing exploratory analysis, I do use my usual tools in a shared environment. Once there’s something that works I may snapshot that and save it in a project.
5 Likes
How do you snapshot an environment?
1 Like
Could Pkg.add perhaps add the latest compatible locally installed version? I want add to add things a.s.a.p., and update can take time to update.
I use separate environments for different scripts, and often use packages with large dependency trees like ModelingToolkit and Makie. When adding these packages to a new environment because I want to quickly try something, I almost always have to wait a long time since usually some of their deps have updated since my last install and it needs to precompile again.
It’s hard for users to think about changing their workflow to avoid more precompilation than they need, but changing Pkg.add behavior would be automatic for everyone. Pkg.status will tell you what is outdated.
10 Likes
I guess I just don’t get why you want to use separate environments for everything if you still want to use all the versions you got from the outer environment. To me this sounds like a request to make Pkg.add worse to support a workflow that doesn’t really make sense (to me at least).
1 Like
I don’t make separate environments for everything, but say if it is a separate project for work, I do.
I don’t know if a Pkg.add that prefers adding already locally available packages is worse. It is faster, and separates updating and adding more clearly. Of course if a package is not available locally it would work as it does now.
That, in your startup.jl file, does that. I has been discussed if that should be the default.
2 Likes
Copy the environment files and then delete things I’m not using in the new copy. At that point I would probably also try updating everything in that environment and make sure it still works.
Isn’t the exact reproducibility of the environment overrated?
I mean, having it is a nice feature. But don’t we actually expect that updating all non breaking versions still keeps everything working?
Maybe there should be another level of “instantiation”, that adds the latest available non breaking versions of the packages compatible with the project, and just warns the user if updates are available. And if the user updates, other environments sharing the same non breaking versions of packages should benefit automatically from that update, upon this level of “instantiation”.
2 Likes
No.
22 Likes
@StefanKarpinski " We’re not even talking about if things install anymore—it’s taken as a given! Anyway, a couple of minutes to install a fully working graphics stack is a delight."
I suggest having this tattooed on the backside of users of lesser programming languages.
2 Likes
No, it doesn’t. It only turns off the registry update. This improves things a bit but doesn’t tell the resolver to use package combinations that are already installed and, ideally, precompiled.
BTW, I have this in my startup.jl for a long time.
2 Likes