I was under the impression that installing a new package should not update any existing packages, unless this is required to satisfy dependencies. Nevertheless, I feel like this is what happens whenever I install anything new. Here’s an example from just now, I installed ApproxFun and got an update to e.g. GPUArrays, which doesn’t seem to be anywhere in ApproxFun’s dependency tree (which I verified by installing ApproxFun into a clean environment, in which case GPUArrays was not installed):
Not an expert here but perhaps GPUArrays is an indirect dependency (i.e. some package on which ApproxFun depends has it as a dependency). This can be checked by inspecting the Manifest.toml for appearences of GPUArrays.
Pkg is free to upgrade indirect dependencies. If you rely on specific versions on indirect dependencies you shouldn’t and should instead add them as explicit dependencies.
GPUArrays is not an indirect dependency of ApproxFun. Like I mentioned above, I checked by installing it into a completely empty environment and noting that GPUArrays was not installed, and also here’s the entry in Manifest.toml where you can see as well:
All indirect dependencies are currently free to upgrade.
Locking GPUArrays from upgrading could have made adding ApproxFun error if it needed a new version of GPUArrays.
I have some ideas of having a “tiered” approach to the resolver where it first tries to resolve with all deps fixed, if that fails it relaxes indirect deps and if that fails, it relaxes direct deps. Haven’t tried it out in practice yet though.
Sorry if I’m being dense, but I don’t understand this. GPUArrays is not in ApproxFun’s direct depencies, nor is it any of its dependencies’ dependencies, etc… So how could locking GPUArrays have made adding ApproxFun error, it doesn’t seem to depend on it in any way?
We don’t know what will end up in the resolved state until we actually run the resolver. You specify the input packages, their allowed version ranges, and get a list of packages at some versions from the resolver.
Sorry, can’t say I know enough about Pkg to understand that response.
As an ignorant Pkg end-user, I still don’t feel like I have the answer to the question of why GPUArrays was updated upon installing ApproxFun in my example, and whether this is expected and/or considered the desired behavior or not.
I just read this thread and I also don’t understand what in the world is going on here… if GPUArrays is not in the transitive dependency closure of ApproxFun, why does Pkg have any business touching GPUArrays when installing or updating ApproxFun?
I would also like to understand what’s going on here. I’m wondering if the cause of GPUArrays being updated could be something like the following:
Let’s say we have two packages Bar and Baz, that both depend on package Foo. Now let’s say you do Pkg.update("Bar") and that causes Foo to be updated to a version that’s incompatible with the current version of Baz. Then Baz will have to be updated even though it’s not a dependency of Bar.
When a single package has a dependency tree several levels deep, I can see how this kind of thing becomes difficult to reason about and you could end up with some changes to seemingly unrelated packages.
In any event, I don’t know how the resolver works so this is entirely speculation.
In short, different versions have different dependencies and figuring out that it is impossible for ApproxFun and GPUArrays to ever “interact” for any set of versions of any allowed solution is not trivial. It’s also about if any of the dependencies of ApproxFun and GPUArrays are shared because in that case their compatability with the shared dependency will interact with each other. So you cannot easily solve for e.g. ApproxFun on its own.
Yes, pretty much. We feed in all the “required” packages (those that we need to be installed because they have been explicitly added) and all the constrains of all the dependencies for all versions, recursively, and the “resolver” gives back a set of dependencies and versions that fulfil these constraints (while trying to keep the versions as high as possible).
This complaint only applies to the old package manager which we are deleting in 1.0.
So it seems like maybe this issue is not on the radar. The documentation says this:
Use update to update an installed package:
(v1.1) pkg> update Example
To update all installed packages, use update without any arguments:
(v1.1) pkg> update
without any explicit mention that all packages will be updated regardless. I think it should explain why it doesn’t work like you would expect or if it is something that will be addressed on the future.
I think this is wrong, though. The package manager probably does try to update only the specified packages, but dependencies work into both directions, so being an indirect dependency of something that indirectly depends on another package may very well lead to an update there.
For example: although GPUArrays and ApproxFun don’t depend on each other, both do depend on LinearAlgebra and AbstractFFTs, possibly with different version requirements between different versions, so an update of ApproxFun also checking GPUArrays is anything but unexpected.
With all due respect, but maybe it is possible to factor out from Pkg only parts that are relevant to actual interaction with git or whatever source is used, and introduce possibility for external packages to implement their own update strategies? I mean, there is LightGraphs.jl and JuMP.jl, so there are packages in Julia ecosystem, that one can use to make rather sophisticated things. In ideal world it could be Pkg.jl with is own solved as a default backend and external packages which can provide other experience.
For reference, the title issue is now solved by the tiered resolver.
] add now has a number of different options for --preserve
Quoting from the docs
Pkg resolves the set of packages in your environment using a tiered approach.
The --preserve command line option allows you to key into a specific tier in the resolve algorithm.
The following table describes the command line arguments to --preserve (in order of strictness).
Argument
Description
all
Preserve the state of all existing dependencies (including recursive dependencies)
direct
Preserve the state of all existing direct dependencies
semver
Preserve semver-compatible versions of direct dependencies
none
Do not attempt to preserve any version information
tiered
Use the tier which will preserve the most version information (this is the default)
So basically now, it will attempt to make a few changes to the currently installed packages as possible in order to install the new package being ] added.
As I understand this:
First it will try and avoid all changes,
then it will try and avoid changing the version of direct dependencies, but allow changes to the version of indirect deprencencies
then it will allow changes to direct dependencies but will avoid incrementing the major (or minor pre-1.0) version, even if Project.toml’s [compat] says it is allowed to
then it will allow changes to everything (as long as permitted by Project.toml’s [compat] section)
then it will error if it still can’t find a compatible set of package that let it add the new one.
If you don’t want that you can set a explict preservation level. ] add Example --preserve=all
so you can get an error if it can’t be added.
Seems to me though that that is relying on the Manifest.toml too much (the Manifest.toml is the thing that knows about what exactly is currently installed, vs the Project.toml that just says what is allows).
I prefer to think of the Manifest.toml as transient, and make sure the version constraints I care about are in the Project.toml
I don’t know what ]up Example does. Docs don’t say it has a --preserve option. I have opened an issue about that.
Just because it would be good to be consistent with add