Installing package seems to update everything unintentionally?

I was under the impression that installing a new package should not update any existing packages, unless this is required to satisfy dependencies. Nevertheless, I feel like this is what happens whenever I install anything new. Here’s an example from just now, I installed ApproxFun and got an update to e.g. GPUArrays, which doesn’t seem to be anywhere in ApproxFun’s dependency tree (which I verified by installing ApproxFun into a clean environment, in which case GPUArrays was not installed):

(v1.1) pkg> add ApproxFun
 Resolving package versions...
 Installed InfiniteArrays ───────────────── v0.1.1
 Installed FillArrays ───────────────────── v0.6.3
 Installed ApproxFunFourier ─────────────── v0.1.1
 Installed BlockBandedMatrices ──────────── v0.4.4
 Installed ApproxFunOrthogonalPolynomials ─ v0.1.1
 Installed ApproxFun ────────────────────── v0.11.1
 Installed ApproxFunBase ────────────────── v0.1.1
 Installed ImageFiltering ───────────────── v0.6.3
 Installed Tables ───────────────────────── v0.2.5
 Installed ZipFile ──────────────────────── v0.8.3
 Installed ToeplitzMatrices ─────────────── v0.5.0
 Installed NamedArrays ──────────────────── v0.9.3
 Installed HierarchicalMatrices ─────────── v0.1.4
 Installed HTTP ─────────────────────────── v0.8.2
 Installed DomainSets ───────────────────── v0.0.2
 Installed WebIO ────────────────────────── v0.8.5
 Installed DualNumbers ──────────────────── v0.6.2
 Installed GPUArrays ────────────────────── v0.7.1
 Installed Conda ────────────────────────── v1.3.0
 Installed CategoricalArrays ────────────── v0.5.4
 Installed DiffEqBase ───────────────────── v5.10.1
 Installed BlockArrays ──────────────────── v0.9.1
 Installed FastGaussQuadrature ──────────── v0.3.3
 Installed ImageCore ────────────────────── v0.8.4
 Installed InvertedIndices ──────────────── v1.0.0
 Installed MatrixFactorizations ─────────── v0.0.4
 Installed CMake ────────────────────────── v1.1.2
 Installed LazyArrays ───────────────────── v0.9.0
 Installed BandedMatrices ───────────────── v0.9.2
 Installed ApproxFunSingularities ───────── v0.1.0
 Installed FastTransforms ───────────────── v0.5.0
 Installed PlotlyBase ───────────────────── v0.2.6
 Installed Nullables ────────────────────── v0.0.8
 Installed ImageMetadata ────────────────── v0.7.1
 Installed LowRankApprox ────────────────── v0.2.3
 Installed FunctionWrappers ─────────────── v1.0.0
  Updating `~/.julia/environments/v1.1/Project.toml`
  [28f2ccd6] + ApproxFun v0.11.1
  Updating `~/.julia/environments/v1.1/Manifest.toml`
  [28f2ccd6] + ApproxFun v0.11.1
  [fbd15aa5] + ApproxFunBase v0.1.1
  [59844689] + ApproxFunFourier v0.1.1
  [b70543e2] + ApproxFunOrthogonalPolynomials v0.1.1
  [f8fcb915] + ApproxFunSingularities v0.1.0
  [aae01518] + BandedMatrices v0.9.2
  [8e7c35d0] + BlockArrays v0.9.1
  [ffab5731] + BlockBandedMatrices v0.4.4
  [631607c0] ↑ CMake v1.1.1 ⇒ v1.1.2
  [324d7699] ↑ CategoricalArrays v0.5.2 ⇒ v0.5.4
  [8f4d0f93] ↑ Conda v1.2.0 ⇒ v1.3.0
  [2b5f629d] ↑ DiffEqBase v5.8.1 ⇒ v5.10.1
  [5b8099bc] + DomainSets v0.0.2
  [fa6b7ba4] + DualNumbers v0.6.2
  [442a2c76] + FastGaussQuadrature v0.3.3
  [057dd010] + FastTransforms v0.5.0
  [1a297f60] ↑ FillArrays v0.6.2 ⇒ v0.6.3
  [069b7b12] + FunctionWrappers v1.0.0
  [0c68f7d7] ↑ GPUArrays v0.7.0 ⇒ v0.7.1
  [cd3eb016] ↑ HTTP v0.8.1 ⇒ v0.8.2
  [7c893195] + HierarchicalMatrices v0.1.4
  [a09fc81d] ↑ ImageCore v0.8.3 ⇒ v0.8.4
  [6a3955dd] ↑ ImageFiltering v0.6.2 ⇒ v0.6.3
  [bc367c6b] ↑ ImageMetadata v0.7.0 ⇒ v0.7.1
  [4858937d] + InfiniteArrays v0.1.1
  [41ab1584] + InvertedIndices v1.0.0
  [5078a376] + LazyArrays v0.9.0
  [898213cb] + LowRankApprox v0.2.3
  [a3b82374] + MatrixFactorizations v0.0.4
  [86f7a689] ↑ NamedArrays v0.9.2 ⇒ v0.9.3
  [4d1e1d77] + Nullables v0.0.8
  [a03496cd] ↑ PlotlyBase v0.2.5 ⇒ v0.2.6
  [bd369af6] ↑ Tables v0.2.4 ⇒ v0.2.5
  [c751599d] + ToeplitzMatrices v0.5.0
  [0f1e0344] ↑ WebIO v0.8.1 ⇒ v0.8.5
  [a5390f91] ↑ ZipFile v0.8.1 ⇒ v0.8.3
  Building WebIO ──→ `~/.julia/packages/WebIO/nNJPv/deps/build.log`
  Building ZipFile → `~/.julia/packages/ZipFile/oD4uG/deps/build.log`
  Building Conda ──→ `~/.julia/packages/Conda/kLXeC/deps/build.log`
  Building CMake ──→ `~/.julia/packages/CMake/nSK2r/deps/build.log`

Is this a bug or am I missing something?

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 1
  JULIA_FFTW_PROVIDER = MKL
5 Likes

Not an expert here but perhaps GPUArrays is an indirect dependency (i.e. some package on which ApproxFun depends has it as a dependency). This can be checked by inspecting the Manifest.toml for appearences of GPUArrays.

Pkg is free to upgrade indirect dependencies. If you rely on specific versions on indirect dependencies you shouldn’t and should instead add them as explicit dependencies.

4 Likes

GPUArrays is not an indirect dependency of ApproxFun. Like I mentioned above, I checked by installing it into a completely empty environment and noting that GPUArrays was not installed, and also here’s the entry in Manifest.toml where you can see as well:

[[ApproxFun]]
deps = ["AbstractFFTs", "ApproxFunBase", "ApproxFunFourier", "ApproxFunOrthogonalPolynomials", "ApproxFunSingularities", "BandedMatrices", "BlockArrays", "BlockBandedMatrices", "Calculus", "DSP", "DomainSets", "DualNumbers", "FFTW", "FastGaussQuadrature", "FastTransforms", "FillArrays", "InfiniteArrays", "IntervalSets", "LazyArrays", "LinearAlgebra", "LowRankApprox", "Random", "RecipesBase", "Reexport", "SparseArrays", "SpecialFunctions", "StaticArrays", "Test", "ToeplitzMatrices"]
git-tree-sha1 = "e960d8b4a1e9762576a14a41bcf06a8e8a832f81"
uuid = "28f2ccd6-bb30-5033-b560-165f7b14dc2f"
version = "0.11.1"

So I still don’t understand why it upgraded it.

1 Like

All indirect dependencies are currently free to upgrade.

Locking GPUArrays from upgrading could have made adding ApproxFun error if it needed a new version of GPUArrays.

I have some ideas of having a “tiered” approach to the resolver where it first tries to resolve with all deps fixed, if that fails it relaxes indirect deps and if that fails, it relaxes direct deps. Haven’t tried it out in practice yet though.

4 Likes

Sorry if I’m being dense, but I don’t understand this. GPUArrays is not in ApproxFun’s direct depencies, nor is it any of its dependencies’ dependencies, etc… So how could locking GPUArrays have made adding ApproxFun error, it doesn’t seem to depend on it in any way?

We don’t know what will end up in the resolved state until we actually run the resolver. You specify the input packages, their allowed version ranges, and get a list of packages at some versions from the resolver.

2 Likes

Sorry, can’t say I know enough about Pkg to understand that response.

As an ignorant Pkg end-user, I still don’t feel like I have the answer to the question of why GPUArrays was updated upon installing ApproxFun in my example, and whether this is expected and/or considered the desired behavior or not.

Short version: It is expected with the current implementation. There are different strategies that could be adopted to make it not happen.

3 Likes

I just read this thread and I also don’t understand what in the world is going on here… if GPUArrays is not in the transitive dependency closure of ApproxFun, why does Pkg have any business touching GPUArrays when installing or updating ApproxFun?

I would also like to understand what’s going on here. I’m wondering if the cause of GPUArrays being updated could be something like the following:

Let’s say we have two packages Bar and Baz, that both depend on package Foo. Now let’s say you do Pkg.update("Bar") and that causes Foo to be updated to a version that’s incompatible with the current version of Baz. Then Baz will have to be updated even though it’s not a dependency of Bar.

When a single package has a dependency tree several levels deep, I can see how this kind of thing becomes difficult to reason about and you could end up with some changes to seemingly unrelated packages.

In any event, I don’t know how the resolver works so this is entirely speculation.

In short, different versions have different dependencies and figuring out that it is impossible for ApproxFun and GPUArrays to ever “interact” for any set of versions of any allowed solution is not trivial. It’s also about if any of the dependencies of ApproxFun and GPUArrays are shared because in that case their compatability with the shared dependency will interact with each other. So you cannot easily solve for e.g. ApproxFun on its own.

6 Likes

Thanks for your explanation! so how is this set of packages to update calculated? Is it like a big constraint solve on the version constraints?

Yes, pretty much. We feed in all the “required” packages (those that we need to be installed because they have been explicitly added) and all the constrains of all the dependencies for all versions, recursively, and the “resolver” gives back a set of dependencies and versions that fulfil these constraints (while trying to keep the versions as high as possible).

4 Likes

If I am understanding correctly, it’s not possible to update a single package due to the approach of the updater algorithm.

This issue describing it was closed two years ago saying:

This complaint only applies to the old package manager which we are deleting in 1.0.

So it seems like maybe this issue is not on the radar. The documentation says this:

Use update to update an installed package:

(v1.1) pkg> update Example

To update all installed packages, use update without any arguments:

(v1.1) pkg> update

without any explicit mention that all packages will be updated regardless. I think it should explain why it doesn’t work like you would expect or if it is something that will be addressed on the future.

1 Like

I think this is wrong, though. The package manager probably does try to update only the specified packages, but dependencies work into both directions, so being an indirect dependency of something that indirectly depends on another package may very well lead to an update there.
For example: although GPUArrays and ApproxFun don’t depend on each other, both do depend on LinearAlgebra and AbstractFFTs, possibly with different version requirements between different versions, so an update of ApproxFun also checking GPUArrays is anything but unexpected.

2 Likes

With all due respect, but maybe it is possible to factor out from Pkg only parts that are relevant to actual interaction with git or whatever source is used, and introduce possibility for external packages to implement their own update strategies? I mean, there is LightGraphs.jl and JuMP.jl, so there are packages in Julia ecosystem, that one can use to make rather sophisticated things. In ideal world it could be Pkg.jl with is own solved as a default backend and external packages which can provide other experience.

For reference, the title issue is now solved by the tiered resolver.

] add now has a number of different options for --preserve
Quoting from the docs

Pkg resolves the set of packages in your environment using a tiered approach.
The --preserve command line option allows you to key into a specific tier in the resolve algorithm.
The following table describes the command line arguments to --preserve (in order of strictness).

Argument Description
all Preserve the state of all existing dependencies (including recursive dependencies)
direct Preserve the state of all existing direct dependencies
semver Preserve semver-compatible versions of direct dependencies
none Do not attempt to preserve any version information
tiered Use the tier which will preserve the most version information (this is the default)

So basically now, it will attempt to make a few changes to the currently installed packages as possible in order to install the new package being ] added.
As I understand this:

  • First it will try and avoid all changes,
  • then it will try and avoid changing the version of direct dependencies, but allow changes to the version of indirect deprencencies
  • then it will allow changes to direct dependencies but will avoid incrementing the major (or minor pre-1.0) version, even if Project.toml’s [compat] says it is allowed to
  • then it will allow changes to everything (as long as permitted by Project.toml’s [compat] section)
  • then it will error if it still can’t find a compatible set of package that let it add the new one.

If you don’t want that you can set a explict preservation level.
] add Example --preserve=all
so you can get an error if it can’t be added.

Seems to me though that that is relying on the Manifest.toml too much (the Manifest.toml is the thing that knows about what exactly is currently installed, vs the Project.toml that just says what is allows).
I prefer to think of the Manifest.toml as transient, and make sure the version constraints I care about are in the Project.toml


I don’t know what ]up Example does. Docs don’t say it has a --preserve option. I have opened an issue about that.
Just because it would be good to be consistent with add

11 Likes