Package compatibility caps

We have heuristically capped the compatibility of many old package versions that we believe not to be compatible with Julia 1.0. The heuristic is:

  • passes PkgEval or recursive dependency thereof => 1.0 compatible
  • has a Julia version interval only containing ≥ 0.7 versions => 1.0 compatible
  • supports Julia ≤ 0.6 & doesn’t use Compat => 1.0 incompatible
  • claims to support ≥ 0.7 & tagged on July 14 or later => 1.0 compatible
  • otherwise => 1.0 incompatible

This isn’t entirely accurate and some packages slip through the cracks and actually work on Julia 1.0 despite the version being older than July 14th and not using Compat, etc. If you have a package version that shouldn’t be capped, the recourse is to make PR like this one which will automatically cause the package to become uncapped. However, please only do this if the package version actually works on Julia 1.0.

There are many packages, however, that still haven’t been updated and are genuinely broken on Julia 1.0; while previously the package manager would happily install these, it will now refuse to do so due to unsatisfiable compatibility constraints. Please just fix the deprecations and make a new release. Don’t use this as an excuse to refactor your code or redesign the package. Just fix the deprecations. They tell you exactly what to do, just do it and then when there are no more warnings and tests pass, tag a new release.

If you are an end user and don’t want to worry about this for now and just want an old uncapped version of the General registry, you can go into ~/.julia/registries/General and do:

git checkout -b dontcapmebro e5748a7e36745cf42952c5451586a79911a524b4

Of course you won’t get any new packages or versions but for today to get things done, this is the easiest way to do it.

21 Likes

Would it be a good idea (and worth the effort?) to inform the package authors via issues?

Are you volunteering to open all those issues?

1 Like

Update: we’ve loosened the cap a bit to cap packages at < 1.0 instead of < 0.7, which means that if you’re using Julia 0.7 there is no cap and you can install all of the packages that you could before, whether they work or not. If you’re using 1.0, however, the cap is there. I have updated my original post to reflect this change.

4 Likes

Nobody should, right?, though I’ll open issues about those cases I stumble on. I do remember automated opening of issues/pr related to pkg infrastructure has been done in the past. In any case, removing the cap for 0.7 seems to take away enough of the pressure, so…

1 Like

Would it make sense for Pkg3 (the new package manager in 1.0) to be restricted to 1.0 compatible packages?

If a package is not compatible with 1.0, why would the 1.0.0 package manager offer it?

Maybe I can help with opening this issue, but what precisely should be done? If just find bunch of packages on GitHub and post some text, that I can do, I always tweaking something when I listing some lectures. But if it something deeper that I probably can’t help.

I will be back to the problem tomorrow now is too late for me.

Could the original poster or someone more knowledgeable in GIT jargon (or whatever this is) define for the average Julia learner what is meant by “cap” in this context? Thanks.

1 Like

It means that instead of the package manager happily installing a package that will be unusable and error when you try to load it, it will error and say that this package does not support 1.0.

2 Likes

This broke Wavelets.jl which is now not anymore installable. This is IMHO not ok that working things are broken in the 1.0 ecosystem.

Is this related to the change?

ipro@~/julia:0:503> julia -v
julia version 1.0.0
ipro@~/julia:0:504> julia
[ Info: startup.jl executed ]
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.0 (2018-08-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Pkg.update()
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PkgTemplates [14b8a8f1]:
 PkgTemplates [14b8a8f1] log:
 ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.3] or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0-0.1.1, 0.2.0-0.2.3]
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left
Stacktrace:
 [1] #propagate_constraints!#61(::Bool, ::Function, ::Pkg.GraphType.Graph, ::Set{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:1005
 [2] propagate_constraints! at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:946 [inlined]
 [3] #simplify_graph!#121(::Bool, ::Function, ::Pkg.GraphType.Graph, ::Set{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:1460
 [4] simplify_graph! at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:1460 [inlined] (repeats 2 times)
 [5] macro expansion at ./logging.jl:311 [inlined]
 [6] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:339
 [7] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1206
 [8] #up#31(::Pkg.Types.UpgradeLevel, ::Pkg.Types.PackageMode, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:184
 [9] #up#30 at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:158 [inlined]
 [10] up at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:154 [inlined]
 [11] #up#27 at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:151 [inlined]
 [12] up() at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:151
 [13] top-level scope at none:0

Yes, PkgTemplates doesn’t load on 1.0.

Sorry about that, wavelets is working on 1.0 (#18243) · JuliaLang/METADATA.jl@f096666 · GitHub should fix so that Wavelets doesn’t get capped anymore on the next registry update.

1 Like

Please tone down the saltiness. As I said, the capping is a heuristic; it will get some packages wrong, which can be fixed on a case-by-case basis. Most of the guesses are correct.

3 Likes

I see. PkgTemplates was a package that something else installed. I am running into trouble with

ulia> Pkg.update()
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Matcha [8d673c98]:
 Matcha [8d673c98] log:
 ├─possible versions are: [0.0.1-0.0.2, 0.1.0-0.1.1] or uninstalled
 ├─restricted by compatibility requirements with Sugar [e18849f4] to versions: [0.0.1-0.0.2, 0.1.0-0.1.1]
 │ └─Sugar [e18849f4] log:
 │   ├─possible versions are: [0.0.1-0.0.3, 0.1.0, 0.2.0, 0.3.0-0.3.1, 0.4.0-0.4.5] or uninstalled
 │   └─restricted to versions * by an explicit requirement, leaving only versions [0.0.1-0.0.3, 0.1.0, 0.2.0, 0.3.0-0.3.1, 0.4.0-0.4.5]
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left

...

julia> Pkg.rm("Matcha")
┌ Warning: `Matcha` not in project, ignoring
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1127
[ Info: No changes

I think I will just wipe what I have installed and start over. it’s not too painful.

I fail to translate this to my mother language but interpret that as that my tone is not ok?

My point is that METADATA is a live system and that this change is breaking things that worked before. My position is that those things should be avoided. I am responsible for a package that is used by people to do real work. The change made my package uninstallable (because Wavelets.jl was uninstallable). What I wish is that the Pkg infrastructure converges into a system with high reliability.

In principle I think that the capping is the right thing (thanks @kristoffer.carlsson for fixing my case) but I at least wanted to give a voice that the Julia users expect a little reliability in METADATA.

At the same time, users also expect that abandonned packages are flagged somehow (this has been asked for multiple times).

It is difficult to do the above without any mistakes in a semi-automated way, and even the latter took time and effort from people, which (to the best of my knowledge) they provided without compensation. Instead of complaining about minor breakages, we should be grateful for their work.

Also, in particular, it seems the issue with Wavelets.jl was fixed super-quickly. I sincerely hope that the people who do real work were not terribly inconvenienced. At the same time, I am curious what kind of work you think the others (including the people who maintain the registry infrastruture) are doing.

7 Likes

It seems I fail to make my point. I have not indicated that others are not doing real work. I am a developer that hangs around in this forum and can fix things. There are many people that can’t do this. Those people do Pkg.update() and have a broken system. Even I had to debug this yesterday for several hours before I found the root cause of it. And then I could not even fix it because action is needed by another package maintainer.

As I have said, this should not be read as a complaint, but I want that people think about whether we want to increase the reliability of the Pkg system or not.

5 Likes

In the future, it might be a good idea to provide a grace period for changes of this nature

2 Likes