Package manager: what are "explicit requirements"?

[yha]

I sometimes run into “Unsatisfiable requirements” error when adding some package A, where the description mentions that some package B has “explicit requirements”. When this happens, I found that removing package B and installing A fixes the problem while automatically installing some version of B (sometimes the same version that was “explicitly required”!).
Although this workaround always works so far, I’d like to understand what is happening. I find the error message pretty confusing as I have never “explicitly required” any specific version. Does it refer to a requirement from some other, unspecified, package? Is uninstalling B really the “standard” way to fix this?

Here is a recent example (where an older version of B==ForwardDiff is eventually installed):

(v1.0) pkg> add JuMP
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReverseDiffSparse [89212889]:
 ReverseDiffSparse [89212889] log:
 ├─possible versions are: [0.0.1-0.0.5, 0.1.0-0.1.13, 0.2.0-0.2.11, 0.3.0, 0.4.0-0.4.1, 0.5.0-0.5.8, 0.6.0, 0.7.0-0.7.3, 0.8.0-0.8.4] or uninstalled
 ├─restricted by compatibility requirements with ForwardDiff [f6369f11] to versions: [0.0.1-0.0.5, 0.1.0-0.1.13, 0.2.0-0.2.11, 0.3.0, 0.4.0-0.4.1] or uninstalled
 │ └─ForwardDiff [f6369f11] log:
 │   ├─possible versions are: [0.0.2-0.0.3, 0.1.0-0.1.8, 0.2.0-0.2.5, 0.3.0-0.3.5, 0.4.0-0.4.2, 0.5.0, 0.6.0, 0.7.0-0.7.5, 0.8.0-0.8.5, 0.9.0, 0.10.0-0.10.1] or uninstalled
 │   └─restricted to versions 0.10.1 by an explicit requirement, leaving only versions 0.10.1
 ├─restricted by julia compatibility requirements to versions: 0.8.2-0.8.4 or uninstalled, leaving only versions: uninstalled
 └─restricted by compatibility requirements with JuMP [4076af6c] to versions: 0.8.0-0.8.4 — no versions left
   └─JuMP [4076af6c] log:
     ├─possible versions are: [0.1.1-0.1.2, 0.2.0, 0.3.0-0.3.2, 0.4.0-0.4.1, 0.5.0-0.5.8, 0.6.0-0.6.3, 0.7.0-0.7.4, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.3, 0.12.0-0.12.2, 0.13.0-0.13.2, 0.14.0-0.14.2, 0.15.0-0.15.1, 0.16.0-0.16.2, 0.17.0-0.17.1, 0.18.0-0.18.5] or uninstalled
     ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.1-0.1.2, 0.2.0, 0.3.0-0.3.2, 0.4.0-0.4.1, 0.5.0-0.5.8, 0.6.0-0.6.3, 0.7.0-0.7.4, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.3, 0.12.0-0.12.2, 0.13.0-0.13.2, 0.14.0-0.14.2, 0.15.0-0.15.1, 0.16.0-0.16.2, 0.17.0-0.17.1, 0.18.0-0.18.5]
     ├─restricted by compatibility requirements with ForwardDiff [f6369f11] to versions: [0.1.1-0.1.2, 0.2.0, 0.3.0-0.3.2, 0.4.0-0.4.1, 0.5.0-0.5.8, 0.6.0-0.6.3, 0.7.0-0.7.4, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.3, 0.18.4-0.18.5] or uninstalled, leaving only versions: [0.1.1-0.1.2, 0.2.0, 0.3.0-0.3.2, 0.4.0-0.4.1, 0.5.0-0.5.8, 0.6.0-0.6.3, 0.7.0-0.7.4, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.3, 0.18.4-0.18.5]
     │ └─ForwardDiff [f6369f11] log: see above
     └─restricted by julia compatibility requirements to versions: 0.18.3-0.18.5 or uninstalled, leaving only versions: 0.18.4-0.18.5

(v1.0) pkg> remove ForwardDiff
  Updating `C:\Users\yharel\.julia\environments\v1.0\Project.toml`
  [f6369f11] - ForwardDiff v0.10.1
  Updating `C:\Users\yharel\.julia\environments\v1.0\Manifest.toml`
 [no changes]

(v1.0) pkg> add JuMP
 Resolving package versions...
 Installed MathProgBase ────── v0.7.7
 Installed ReverseDiffSparse ─ v0.8.4
 Installed JuMP ────────────── v0.18.5
  Updating `C:\Users\yharel\.julia\environments\v1.0\Project.toml`
  [4076af6c] + JuMP v0.18.5
  Updating `C:\Users\yharel\.julia\environments\v1.0\Manifest.toml`
  [f6369f11] ↓ ForwardDiff v0.10.1 ⇒ v0.9.0
  [4076af6c] + JuMP v0.18.5
  [fdba3010] + MathProgBase v0.7.7
  [89212889] + ReverseDiffSparse v0.8.4

[kristoffer.carlsson]

I think (I am not 100% familiar with the resolver) that an explicit requirement is a requirement that comes from the package being in your Project file. So it is not in the resolver because anyone depends on it, it is because you explicitly wanted it (by adding the package).

[kevbonham]

@kristoffer.carlsson

So it is not in the resolver because anyone depends on it, it is because you explicitly wanted it (by adding the package).

To expand on this - when you add a package, it adds a specific version of that package. If you don’t update for a while, that package might fall behind and be too old for some new packages’ deps. Is that right?

In other words, rather than removing FowardDiff, you may have just been able to update it. I think…

[baggepinnen]

I’m having similar issues with package requirements. This time it’s StatsBase that causes issues. I also find the message
restricted to versions 0.30.0 by an explicit requirement, leaving only versions 0.30.0 kind of cryptic. Where does this explicit requirement come from and how can I work around the problem. I have not myself set it, so it would be nice if the error message could give me some more pointers. I have tried ] up and the error persists. The full error message is

ERROR: Unsatisfiable requirements detected for package StatsBase [2913bbd2]:
 StatsBase [2913bbd2] log:
 ├─possible versions are: [0.1.0, 0.2.0-0.2.1, 0.2.3-0.2.10, 0.3.0-0.3.13, 0.4.0-0.4.4, 0.5.0-0.5.3, 0.6.0-0.6.16, 0.7.0-0.7.4, 0.8.0-0.8.3, 0.9.0, 0.10.0, 0.11.0-0.11.1, 0.12.0, 0.13.0-0.13.1, 0.14.0-0.14.1, 0.15.0, 0.16.0-0.16.1, 0.17.0, 0.18.0, 0.19.0-0.19.5, 0.20.0-0.20.1, 0.22.0, 0.23.0-0.23.1, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0-0.28.1, 0.29.0, 0.30.0] or uninstalled
 ├─restricted to versions 0.30.0 by an explicit requirement, leaving only versions 0.30.0
 └─restricted by compatibility requirements with StatsPlots [f3b207a7] to versions: [0.1.0, 0.2.0-0.2.1, 0.2.3-0.2.10, 0.3.0-0.3.13, 0.4.0-0.4.4, 0.5.0-0.5.3, 0.6.0-0.6.16, 0.7.0-0.7.4, 0.8.0-0.8.3, 0.9.0, 0.10.0, 0.11.0-0.11.1, 0.12.0, 0.13.0-0.13.1, 0.14.0-0.14.1, 0.15.0, 0.16.0-0.16.1, 0.17.0, 0.18.0, 0.19.0-0.19.5, 0.20.0-0.20.1, 0.22.0, 0.23.0-0.23.1, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0-0.28.1, 0.29.0] — no versions left
   └─StatsPlots [f3b207a7] log:
     ├─possible versions are: 0.10.0-0.10.2 or uninstalled
     └─restricted to versions * by an explicit requirement, leaving only versions 0.10.0-0.10.2

My workaround that works locally, but not on travis, was

1. rm StatsBase
2. rm Distributions
3. add StatsPlots
4. add Distributions
5. add StatsBase
   which resulted in

(LowLevelParticleFilters) pkg> st
Project LowLevelParticleFilters v0.1.0
    Status `~/.julia/dev/LowLevelParticleFilters/Project.toml`
  [31c24e10] + Distributions v0.18.0
  [50d2b5c4] + Lazy v0.13.2
  [90014a1f] + PDMats v0.9.6
  [d96e819e] + Parameters v0.10.3
  [90137ffa] + StaticArrays v0.10.3
  [2913bbd2] + StatsBase v0.29.0
  [f3b207a7] + StatsPlots v0.10.2
  [6310b701] + Yeppp v0.4.0
  [37e2e46d] + LinearAlgebra 
  [9a3f8284] + Random 
  [10745b16] + Statistics 

It would be very nice to know that the explicit requirement comes from me doing add StatsBase (which added v0.30.0) before I added the other packages (one of which required <= v0.29.0), and that this workaround existed.

[baggepinnen]

I really can’t figure out how to work around this on travis. The package now passes tests locally after some rm/add gymnastics, but the travis build fails due to the explicit requirement, even though I explicitly added compatibility with more versions in the Project.toml. The build log is here and the Project.toml is here. Can anyone help me understand what I am doing wrong, or if there is a bug somewhere in the resolver etc.?

[kristoffer.carlsson]

Here, it basically means that one of the test dependencies (Plots) is not compatible with the versions resolved for the package (which includes StatsBase 0.30).

https://github.com/JuliaRegistries/General/pull/227 seems to include Plots to be compatible with StatsBase 0.30.

[baggepinnen]

Oki thanks. Unfortunately, I still do not understand how the resolver came to say restricted to versions 0.30.0 by an explicit requirement, leaving only versions 0.30.0, could it be made to say where the explicit requirement came from?

[kristoffer.carlsson]

It is basically

(Foo) pkg> st
    Status `C:\Users\Kristoffer\Foo\Project.toml`
  [2913bbd2] StatsBase v0.30.0

(Foo) pkg> add Plots
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Plots [91a5bcdd]:
 Plots [91a5bcdd] log:
...
│ └─StatsBase [2913bbd2] log:
    └─restricted to versions 0.30.0 by an explicit requirement, leaving only versions 0.30.0

Pkg doesn’t downgrade packages in the Project file.

(Foo) pkg> rm StatsBase
...

(Foo) pkg> add StatsBase Plots
 Resolving package versions...
  Updating `C:\Users\Kristoffer\Foo\Project.toml`
  [91a5bcdd] + Plots v0.24.0
  [2913bbd2] + StatsBase v0.29.0

[baggepinnen]

So me adding

[compat]
StatsBase = "0.29,0.30"

in Project.toml is not the way to go to say that this particular package supports also v0.29?

[kristoffer.carlsson]

The problem is not that the package itself does not support 0.29, it does. The problem is that a test-dependency (Plots) does not support the version that the package would use if that test dependency was not there.

It is basically this issue: https://github.com/JuliaLang/Pkg.jl/issues/110.

[baggepinnen]

Okay, I somewhat understand after reading the discussion in the issue, thanks. So if I would like my tests to pass, the only option I have other than waiting for dependencies to update, is to say that I only support StatsBase v0.29 for now, since Plots only support v0.29 at the moment?

[kristoffer.carlsson]

You can also check in a Manifest with StatsBase 0.29 installed.

[kristoffer.carlsson]

But I merged the PR now so it should work now.

[tkf]

]add Plots and ]test Plots in a clean Julia install still does not work (checked it in a clean Docker). I think this is because https://github.com/JuliaRegistries/General/pull/227 does not change StatsPlots’s compat info. Since StatsPlots does not claim any compat information for StatsBase, I believe the correct behavior is to let it be installed even though the test may fail. I wrote a more detailed explanation for why I thought this was the correct behavior in Dependency problem with FIllArrays - #41 by tkf.

[kristoffer.carlsson]

This is an interaction between the upper bounds placed by the METADATA converter and https://github.com/JuliaLang/Pkg.jl/issues/110.

As it is right now, if a test dependency is not compatible with the versions of packages that gets resolved without the test dependency, the package will fail with a resolution error when tested.

The workaround is to check in a Manifest with versions that are compatible with the test dependencies. We should maybe think more about https://github.com/JuliaLang/Pkg.jl/issues/110 and loosen how hard the version of project dependencies are being kept.

[Tamas_Papp]

@kristoffer.carlsson

The workaround is to check in a Manifest with versions that are compatible with the test dependencies.

My understanding about the practicalities of checking in the Manifest.toml into the repo is the following, is this correct?

1. for users of the package, who installed it via pkg> add or pkg> dev, the Manifest.toml in the package directory is ignored entirely, so this is innocuous,

2. it is used for pkg> test ThatPackage, so then it is useful (the only harm is that it may fetch some earlier versions of various packages, but again that is fine),

3. for CI, it will use that exact test environment, which has some benefits, but also negates some of the advantages of CI (testing with package versions which I don’t happen to have installed on my computer). This can be worked around by doing a Pkg.update() in the CI script; if it works it is fine, if it breaks then the manifest works as a backup.

[kristoffer.carlsson]

1. Yes, the manifest is only “active” if the project for that package is the current Project for example by starting with --project in that directory.

2. It is used when running test on the current active project (which is the case in CI). It is not used for test ThatPackage unless ThatPackage is the current active project.

3. For CI, if a manifest is checked in, it will be used (since the active project will be that of the package).

[Tamas_Papp]

Thanks. With the manifest checked in, is it a good idea to do a

Pkg.update()

then in the CI (eg Travis) script? My concern is that I may forget to keep updating the manifest regularly and could be testing a configuration that is no longer relevant as packages moved on.

[tkf]

I think you misunderstand my point. The resolver is a different issue and I’m not asking for a “user-space” workaround.

@kristoffer.carlsson

the upper bounds placed by the METADATA converter

If I understand correctly, this is a wrong implementation of the specification of REQUIRE file. I understand that it didn’t matter before the full transition to General but I think the current setup of General ignores what was specified in REQUIRE files.

Of course, I don’t know much about packaging system implementation so I may be missing some very obvious. I hope I’m not making unnecessary noise.

[kristoffer.carlsson]

@tkf

I think you misunderstand my point. The resolver is a different issue and I’m not asking for a “user-space” workaround.

I’m not sure if you are replying to me but anyway, I just pointed out the technical reason why test Plots fail. If you didn’t find it useful, perhaps someone else will.