JULIA_PKG_PRESERVE_TIERED_INSTALLED is mentioned in the docs for ] add.
But this is not the only way packages get installed.
A particular case I am interested in is resolve.
If you delete the Manifest.toml and run resolve it will install all the dependancies.
I want to know if setting JULIA_PKG_PRESERVE_TIERED_INSTALLED will actually work to make it select ones i have installed for some other project or not.
I feel like it should.
It’s really hard to test this, and I haven’t been able to find where in the source code ] resolve decided to install things, to see what options it uses.
It’s queried via default_preserve(), which is called iff add or develop are called. I am not especially familiar with that part of resolve, but I think the answer is: no.
However, I wouldn’t even know how it could preserve the versions after the manifest has been deleted. In my book deleting the manifest means: “Start from scratch – I don’t care whether there is any similarity to what we had before”. So I think this combination would depend on you not deleting the manifest.
In general it’s my experience that the more exotic rarely tested combinations of Pkg.jl typically don’t work as you’d hope. Instead of complaining I started to create a PR whenever I identified such a shortcoming in our use cases and I encourage everyone to do the same.
That being said, I think we have a serious lack of review time in Julia, which especially shows there. As reviewing is not the favorite activity for most developers, I am not sure whether we can solve this without having people doing this during their paid time, especially as the imbalance increased with LLMs making it much easier to create code, but not so much easer to review code.
Anyway: If you have commit permissions to accept PRs in JuliaLang, please consider reviewing some Pkg.jl PRs. A nearly three digit number of open PRs is definitely too much for a project of that size (around 1 open PR per 200 lines of regular code).
JULIA_PKG_PRESERVE_TIERED_INSTALLED has 2 parts to it.
1 is the preserve which as you say needs the manifest, because it wants to minimise changes to preserve what is in the manifest.
But the other is the installed which doesn’t depend on the manifest, it depends on what else is already installed on your system.
And that’s actually the feature i am interested in.
Ah, I see. Thanks for the explanation. Yes, this could work. Although a ton of questions need to be answered (What if multiple versions of a package are installed in other environments? What if installed versions of different packages across different environments are incompatible to each other?).
May I ask what the use case is? I am wildly speculating, but it is too much fun not to do so : It does not seem to be “get the (hopefully) most bug-free, because most-recent version”. It does also not seem to be “get the package versions with known bugs in their interplay”. It rather sounds like “I have a really slow downstream and I don’t care which bugs I get, I just want to get this up and running as quickly as possible”. But then again why not start from a hopefully known-good manifest?
Ah, I see. Thanks for the explanation. Yes, this could work. Although a ton of questions need to be answered (What if multiple versions of a package are installed in other environments? What if installed versions of different packages across different environments are incompatible to each other?).
Presumably those are all already determines by the behavour of JULIA_PKG_PRESERVE_TIERED_INSTALLED with ]add.
May I ask what the use case is?
The use case two things.
I am mostly using very well established features of packages, but i am using them within a huge dependancy tree. Most of the stuff I am doing depends on Catalyst.jl, which has 153 dependancies, and Plots.jl which has 167 indirect dependancies. (and together they have 310 net). That is 10-20 minutes pre-compile time. I often want to create a little enviroment (maybe with ]activate --temp) to test something out, or make a MWE. I would like to minimise my precompilation time, I don’t want to get the latest feature i won’t use from some dependenancy of a dependancy.
As a hack around the fact that pyjuliapkg does not support Manifest.toml for the julia enviroment it creates (equivalent of Manifest.toml / pixi.lock · Issue #78 · JuliaPy/pyjuliapkg · GitHub). So if anything indirect gets a release and the enviroment needs to be resolved (which i think is needed all the time on CI) we install it, which voids the precompilation cache. if the resolving was to do as per preserve installed tiered, then it would preferentially use installed packages (and on CI julia-cache does cache not just compilations but also installation). And the latest installed packages would generally end up being a set that was precompiles together last time, before that release.
Thanks for the explanations. Indeed, interesting use cases and I can now totally see where you are coming from.
Have you considered using stacked environments together with a temporary environment? As far as I know they only work in REPL-only workflows, but I guess this is the typical workflow for a temporary environment anyway.
I think using resolve on CI is at least problematic and it helps to think about the implications.
For app packages (anything which is meant to be started directly and not included into something else, so not limited to what Pkg calls “app”) you typically should check-in a manifest and CI should instantiate instead of resolve.
For non-app packages it’s complicated:
If they are currently only used from within one repo, I think using a monorepo and a workspace is the cleanest solution, so they are covered by the manifest with instantiate, too.
If this is not possible and they are in their own repo, you can still decide to get a manifest from somewhere and use that.
If you also don’t want to do this, you actually need to resolve and be aware that CI is now no longer reproducible, meaning the success of a CI run depends both on your CI setup (including your repo) and on the dependencies, meaning you can have a totally random looking pass-fail pattern between your runs, because it is determined by changes in your (direct and indirect) dependencies.
Have you considered using stacked environments together with a temporary environment? As far as I know they only work in REPL-only workflows, but I guess this is the typical workflow for a temporary environment anyway.
They actually work everywhere. As far as they work at all.
They are a pretty dodgy feature that that does not actually play nicely at all with anything in Pkg.
They ignore compat bounds conflicts between layers.
And they violate the principle that your Manifest.toml should describe your dependancies.
But they are pretty useful so we still have them.
But thankfully almost noone uses them outside the case of the per-minor version global enviroment.
: It does not seem to be “get the (hopefully) most bug-free, because most-recent version”. It does also not seem to be “get the package versions with known bugs in their interplay”. It rather sounds like “I have a really slow downstream and I don’t care which bugs I get, I just want to get this up and running as quickly as possible”. But then again why not start from a hopefully known-good manifest?