Why MYPACKAGE is forcing other package downgrade (with an unbounded require)?

sylvaticus · January 8, 2018, 5:47am

Hello I develop a package (OdsIO) whose require is:

julia 0.5
PyCall 1.7
DataFrames 0.8.0
DataStructures 0.5.0

I have read the require file format specifications, and the format I use should indicate any version higher than X. This is the intended behaviour.

I has been notified however that my package make downgrading other packages.
On a new Julia 0.6 installation (note the various downgrading when package OdsIO is added):


julia> Pkg.update()
INFO: Initializing package repository /home/lobianco/.julia/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
INFO: Updating METADATA...
INFO: Computing changes...
INFO: No packages to install, update or remove

julia> Pkg.add("CSV")
INFO: Updating cache of BinDeps...
INFO: Cloning cache of CSV from https://github.com/JuliaData/CSV.jl.git
INFO: Updating cache of CategoricalArrays...
INFO: Cloning cache of CodecZlib from https://github.com/bicycle1885/CodecZlib.jl.git
INFO: Updating cache of Compat...
INFO: Updating cache of DataFrames...
INFO: Cloning cache of DataStreams from https://github.com/JuliaData/DataStreams.jl.git
INFO: Updating cache of DataStructures...
INFO: Cloning cache of Missings from https://github.com/JuliaData/Missings.jl.git
INFO: Updating cache of Reexport...
INFO: Updating cache of SHA...
INFO: Updating cache of SortingAlgorithms...
INFO: Updating cache of SpecialFunctions...
INFO: Updating cache of StatsBase...
INFO: Cloning cache of TranscodingStreams from https://github.com/bicycle1885/TranscodingStreams.jl.git
INFO: Updating cache of URIParser...
INFO: Cloning cache of WeakRefStrings from https://github.com/quinnj/WeakRefStrings.jl.git
INFO: Installing BinDeps v0.8.2
INFO: Installing CSV v0.2.0
INFO: Installing CategoricalArrays v0.3.3
INFO: Installing CodecZlib v0.4.2
INFO: Installing Compat v0.43.0
INFO: Installing DataFrames v0.11.3
INFO: Installing DataStreams v0.3.4
INFO: Installing DataStructures v0.7.4
INFO: Installing Missings v0.2.4
INFO: Installing NamedTuples v4.0.0
INFO: Installing Reexport v0.1.0
INFO: Installing SHA v0.5.2
INFO: Installing SortingAlgorithms v0.2.0
INFO: Installing SpecialFunctions v0.3.7
INFO: Installing StatsBase v0.19.5
INFO: Installing TranscodingStreams v0.4.1
INFO: Installing URIParser v0.3.0
INFO: Installing WeakRefStrings v0.4.1
INFO: Building SpecialFunctions
INFO: Package database updated

julia> Pkg.add("PyCall")
INFO: Updating cache of Conda...
INFO: Updating cache of JSON...
INFO: Cloning cache of Nullables from https://github.com/JuliaArchive/Nullables.jl.git
INFO: Installing Conda v0.7.1
INFO: Installing JSON v0.16.3
INFO: Installing MacroTools v0.4.0
INFO: Installing Nullables v0.0.2
INFO: Installing PyCall v1.15.0
INFO: Building Conda
INFO: Building PyCall
INFO: PyCall is using python (Python 2.7.12) at /usr/bin/python, libpython = libpython2.7
INFO: /home/lobianco/.julia/v0.6/PyCall/deps/deps.jl has been updated
INFO: /home/lobianco/.julia/v0.6/PyCall/deps/PYTHON has been updated
INFO: Package database updated

julia> Pkg.add("DataStructures")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.add("OdsIO")
INFO: Updating cache of FileIO...
INFO: Downgrading CSV: v0.2.0 => v0.1.5
INFO: Downgrading CategoricalArrays: v0.3.3 => v0.1.6
INFO: Installing DataArrays v0.6.2
INFO: Downgrading DataFrames: v0.11.3 => v0.10.1
INFO: Downgrading DataStreams: v0.3.4 => v0.1.3
INFO: Installing FileIO v0.6.1
INFO: Installing GZip v0.3.0
INFO: Installing NullableArrays v0.1.2
INFO: Installing OdsIO v0.2.0
INFO: Downgrading WeakRefStrings: v0.4.1 => v0.2.0
INFO: Removing CodecZlib v0.4.2
INFO: Removing Missings v0.2.4
INFO: Removing NamedTuples v4.0.0
INFO: Removing TranscodingStreams v0.4.1
INFO: Building SpecialFunctions
INFO: Building Conda
INFO: Building PyCall
INFO: PyCall is using /usr/bin/python (Python 2.7.12) at /usr/bin/python, libpython = libpython2.7
INFO: /home/lobianco/.julia/v0.6/PyCall/deps/deps.jl has been updated
INFO: /home/lobianco/.julia/v0.6/PyCall/deps/PYTHON has not changed
INFO: Building OdsIO
INFO: Package database updated

Why is that? Which is considered the right approach? I indicated a minimum version on each package because that is the one I did tested my package with and “guarantee” that works.

Liso · January 8, 2018, 1:23pm

Out of topic:
You are using pip install independently on PyCall.conda. Other package could install lxml using conda install.

(and conda-forge channel has ezodf too: Ezodf :: Anaconda.org )

In case PyCall.python is distro’s python (default on Linux) you probably overwrite distro’s lxml package! (also if you’ll use conda install)

BTW ezodf requires lxml so you probably don’t need to install lxml explicily!

Maybe somebody (@stevengj ?) could tell what is preferable way to install packages into python sub-eco-system? How to avoid dependency hell?

kristoffer.carlsson · January 8, 2018, 2:04pm

Try bump your julia requirement to 0.6.

Edit: I don’t think the above makes any sense.

nalimilan · January 9, 2018, 10:17pm

This is most likely due to one of the dependencies of your package not supporting DataFrames 0.11 yet. It’s not obvious to me which one is the culprit, but you could try adding them one by one on a fresh install after installing DataFrames and see when downgrading happens. Then repeat the operation with the dependencies of the problematic dependency, and so on.

carlobaldassi · January 10, 2018, 1:24am

The real reason is that the package requirements were changed in METADATA by this commit when DataFrames 0.11 was tagged (not only for your package, but for all of them; as a cautionary measure, I suppose — @nalimilan, can you confirm?)

So when you do Pkg.add("OdsIO") the “requires” file in METADATA is used, overriding the one of the package. Currently, if you check out the package it’s still seen as v"0.2.0" and its requirements overridden with the ones in METADATA, but if you then add whatever commit to it, its version will be seen as v"0.2.0+" and the package manager will then use the “REQUIRE” file in the package directory.

Anyway, if the package is compatible with DataFrames v0.11 then the right solution is just to remove the upper bound from METADATA.

nalimilan · January 10, 2018, 9:36am

You’re right @carlobaldassi. The systematic addition of upper bounds on DataFrames 0.11 isn’t ideal, but that’s currently the only way to ensure users do not end up with broken packages after calling Pkg.update(). Hopefully we’ll have a clearer way of handling this with Pkg3.