How to find package preventing others from updating

packages
package-manager

#1

While trying to upgrade a package I found that some other package held it back with an upper version bound. To track down the offending package, I found the following function useful

function find_holdback(held_back_package, recursive = 0, installed = collect(keys(Pkg.installed())), inset=0)
    @assert held_back_package ∈ installed "The package you are querying is not installed"
    for package in installed
        REQUIRE = Pkg.dir(package, "REQUIRE")
        isfile(REQUIRE) || continue
        for line in eachline(REQUIRE)
            splitline = split(line, ' ')
            splitline[1] == held_back_package || continue
            for i = 1:inset print("    ") end
            if length(splitline) > 2
                print_with_color(:red, @sprintf("Package %-35s requires %s\n", package, line))
            else
                @printf("Package %-35s requires %s\n", package, line)
            end
            recursive > 0 && find_holdback(package, recursive-1, installed, inset+1)
        end
    end
end

julia> find_holdback("ForwardDiff", 2) # Prints requires with a depth of 2

Example output:

julia> find_holdback("ForwardDiff", 2)
Package NLsolve                                            requires ForwardDiff 0.3.0
    Package SteadyStateDiffEq                                  requires NLsolve
        Package DifferentialEquations                              requires SteadyStateDiffEq 0.0.1
    Package DiffEqCallbacks                                    requires NLsolve
        Package DifferentialEquations                              requires DiffEqCallbacks 0.0.2
    Package OrdinaryDiffEq                                     requires NLsolve 0.9.1
        Package DelayDiffEq                                        requires OrdinaryDiffEq 1.6.0
        Package DifferentialEquations                              requires OrdinaryDiffEq 2.0.0
    Package StochasticDiffEq                                   requires NLsolve
        Package DifferentialEquations                              requires StochasticDiffEq 2.0.0
    Package FiniteElementDiffEq                                requires NLsolve 0.7.3
        Package DifferentialEquations                              requires FiniteElementDiffEq 0.3.0
Package ReverseDiffSparse                                  requires ForwardDiff 0.3 0.5
    Package JuMP                                               requires ReverseDiffSparse 0.7 0.8
Package Optim                                              requires ForwardDiff 0.3.0 0.5.0
    Package NLsolve                                            requires Optim
        Package SteadyStateDiffEq                                  requires NLsolve
        Package DiffEqCallbacks                                    requires NLsolve
        Package OrdinaryDiffEq                                     requires NLsolve 0.9.1
        Package StochasticDiffEq                                   requires NLsolve
        Package FiniteElementDiffEq                                requires NLsolve 0.7.3
    Package DiffEqParamEstim                                   requires Optim
        Package DifferentialEquations                              requires DiffEqParamEstim 0.3.0
    Package KernelDensity                                      requires Optim
        Package StatPlots                                          requires KernelDensity
Package ReverseDiff                                        requires ForwardDiff 0.3.4 0.5.0
    Package LTVModels                                          requires ReverseDiff
        Package GuidedPolicySearch                                 requires LTVModels
Package DiffEqCallbacks                                    requires ForwardDiff 0.3.0
    Package DifferentialEquations                              requires DiffEqCallbacks 0.0.2
Package DiffEqParamEstim                                   requires ForwardDiff
    Package DifferentialEquations                              requires DiffEqParamEstim 0.3.0
Package Roots                                              requires ForwardDiff 0.2.0
    Package Sundials                                           requires Roots
        Package DifferentialEquations                              requires Sundials 0.11.0
    Package OrdinaryDiffEq                                     requires Roots 0.2.1
        Package DelayDiffEq                                        requires OrdinaryDiffEq 1.6.0
        Package DifferentialEquations                              requires OrdinaryDiffEq 2.0.0
    Package StochasticDiffEq                                   requires Roots
        Package DifferentialEquations                              requires StochasticDiffEq 2.0.0
Package OrdinaryDiffEq                                     requires ForwardDiff 0.2.4
    Package DelayDiffEq                                        requires OrdinaryDiffEq 1.6.0
        Package DifferentialEquations                              requires DelayDiffEq 0.3.0
    Package DifferentialEquations                              requires OrdinaryDiffEq 2.0.0
Package Flux                                               requires ForwardDiff
Package JuMP                                               requires ForwardDiff 0.3 0.5
Package StochasticDiffEq                                   requires ForwardDiff
    Package DifferentialEquations                              requires StochasticDiffEq 2.0.0

How to upgrade from DataFrames 0.10.1 to 0.11.3?
GLM.jl module : 1 test failed
Cannot add DifferentialEquations
Issue with ExcelReaders-DataFrames interaction
Count number of substrings in dataframe column
#2

Great stuff! Thanks for sharing.


#3

I like this, but IIUC your approach is not quite right when an upper bound is introduced via METADATA. This occurred, for example, when the recent breaking revision of DataFrames forced them to bound all dependencies. You could handle these cases by getting the lines from METADATA/$package/versions/$version/requires file instead of the package’s REQUIRE file.


How to upgrade from DataFrames 0.10.1 to 0.11.3?
#4

just in case someone finds this useful, here is an alternative proposed by @Tamas_Papp:

➜  FredData git:(master) grep DataFrames ~/.julia/v0.6/**/REQUIRE
/Users/74097/.julia/v0.6/DataFramesMeta/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/Fertility/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/FredData/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/GLM/test/REQUIRE:DataFrames 0.9.0
/Users/74097/.julia/v0.6/JLD/test/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/MomentOpt/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/MomentOpt/REQUIRE:DataFramesMeta
/Users/74097/.julia/v0.6/MomentOpt/test/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/QuantileRegression/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/Query/test/REQUIRE:DataFrames
/Users/74097/.julia/v0.6/RData/REQUIRE:DataFrames 0.9
/Users/74097/.julia/v0.6/StatsBase/test/REQUIRE:DataFrames

#5

This looks like a really handy function, but it doesn’t seem to be helping me - I wonder if anyone can see what’s going on? I want the latest version of RCall (0.9.0, soon to be 0.10.0), but it’ll only give me 0.8.1. Installing RCall, I see:

julia> Pkg.update()
INFO: Updating METADATA...
INFO: Updating ArchGDAL master...
INFO: Updating JuliaDB master...
INFO: Updating ECCodes master...
INFO: Updating Phylo master...
INFO: Updating AxisArrays master...
INFO: Updating Distances master...
INFO: Updating ECCodesExamples master...
INFO: Updating Unitful master...
INFO: Updating GDAL master...
INFO: Updating Junet master...
WARNING: Package Diversity: skipping update (dirty)...
WARNING: Package FluData: skipping update (dirty)...
INFO: Computing changes...
INFO: No packages to install, update or remove

julia> Pkg.add("RCall")
INFO: Installing NamedArrays v0.7.0
INFO: Installing RCall v0.8.1
INFO: Building SpecialFunctions
INFO: Package database updated

julia> find_holdback("RCall", 2)

julia> Pkg.status("RCall")
 - RCall                         0.8.1
julia> Pkg.checkout("RCall")
INFO: Checking out RCall master...
INFO: Pulling RCall latest master...
ERROR: resolve is unable to satisfy package requirements.
  The problem was detected when trying to find a feasible version
  for package CategoricalArrays.
  However, this only means that package CategoricalArrays is involved in an
  unsatisfiable or difficult dependency relation, and the root of
  the problem may be elsewhere.

Stacktrace:
 [1] resolve(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at ./pkg/resolve.jl:48
...

Suggesting there’s some problem with dependencies for CategoricalArrays, but when I use find_holdback() again, I just get:

julia> find_holdback("CategoricalArrays", 2)
Package DataTables                                         requires CategoricalArrays 0.1.2
Package Feather                                            requires CategoricalArrays 0.0.6
Package RCall                                              requires CategoricalArrays 0.1.0
Package DataStreams                                        requires CategoricalArrays 0.0.5
    Package DataTables                                         requires DataStreams 0.1.0
    Package ArchGDAL                                           requires DataStreams
    Package Feather                                            requires DataStreams 0.1.0

And nothing has an upper bound on CategoricalArrays. In fact searching through the REQUIRE files, I see almost nothing with an upper bound on anything any longer (LearnBase and IntervalTrees for some reason). Any suggestions?


#6

DataTables (at least) requires an old CategoricalArrays version.


#7

Thanks. According to my .julia/v0.6 folder, DataTables has a pretty simple:

CategoricalArrays 0.1.2

requirement with no upper limit (as suggested by find_holdback()). However, I’ve just deleted the whole folder and reinstalled some things and the problem seems to have gone away. I haven’t reinstalled everything, but touch wood the problem has gone away…


#8

I think the problem is that the package manager uses the information present in METADATA, which doesn’t necessarily reflect the one returned by the script, which looks at each package’s directory. In the present case, we have added upper bounds to METADATA, and updated DataTables master, but no release has been tagged.


#9

Ah, okay. Perfect. Adding DataTables makes everything fall apart. That clarifies everything. I’d forgotten that METADATA can be changed without changing the underlying packages.


#10

Is there a way to find all held back packages (instead of the packages holding one package back)?