Seemingly incorrect dependencies resolution in Pkg

rdyro · October 13, 2019, 1:14am

I’m trying to develop Zygote, but can’t successfully install the latest development version from Github. When I try installing the package from the Github repo directly or trying to use the dev command with it, I get the following error:

julia> versioninfo()
Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

(v1.2) pkg> add Zygote
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Zygote ─ v0.3.4
  Updating `~/.julia/environments/v1.2/Project.toml`
  [e88e6eb3] + Zygote v0.3.4
  Updating `~/.julia/environments/v1.2/Manifest.toml`
 [no changes]

(v1.2) pkg> dev Zygote
  Updating git-repo `https://github.com/FluxML/Zygote.jl.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

shell> git clone https://github.com/FluxML/Zygote.jl.git
Cloning into 'Zygote.jl'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 5709 (delta 7), reused 15 (delta 0), pack-reused 5681
Receiving objects: 100% (5709/5709), 1012.82 KiB | 4.25 MiB/s, done.
Resolving deltas: 100% (3810/3810), done.

(v1.2) pkg> add ./Zygote.jl
   Cloning git-repo `./Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

(v1.2) pkg>

ZygoteRules is available on Github in version 0.2.0, but in the general registry only up to 0.1.0. Zygote’s Github manifest correctly specifies that ZygoteRules should be taken from a Github link and not the general registry, but the package manager does not seem to notice.

Now, maybe this isn’t the right way to go, but I tried using activate, to make sure Zygote’s manifest is used as is, but this is the result.

shell> git clone https://github.com/FluxML/Zygote.jl.git
Cloning into 'Zygote.jl'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 5709 (delta 7), reused 15 (delta 0), pack-reused 5681
Receiving objects: 100% (5709/5709), 1012.82 KiB | 3.99 MiB/s, done.
Resolving deltas: 100% (3810/3810), done.

(v1.2) pkg> add Zygote.jl/
   Cloning git-repo `Zygote.jl/`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
 Resolving package versions...
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `~/.julia/registries/General`
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

(v1.2) pkg> activate Zygote.jl/
Activating environment at `~/Dropbox/projects/***/Zygote.jl/Project.toml`

(Zygote) pkg> add Zygote.jl/
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
ERROR: UndefVarError: mode not defined
Stacktrace:
 [1] #add#25(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:96
 [2] add at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:69 [inlined]
 [3] do_add!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:496
 [4] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:790
 [5] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:789
 [6] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:403
 [7] #do_cmd#23(::Bool, ::typeof(Pkg.REPLMode.do_cmd), ::REPL.LineEditREPL, ::String) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:382
 [8] do_cmd at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:378 [inlined]
 [9] (::getfield(Pkg.REPLMode, Symbol("##28#31")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:610
 [10] #invokelatest#1 at ./essentials.jl:790 [inlined]
 [11] invokelatest at ./essentials.jl:789 [inlined]
 [12] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/LineEdit.jl:2306
 [13] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:1038
 [14] run_repl(::REPL.AbstractREPL, ::Any) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:201
 [15] (::getfield(Base, Symbol("##737#739")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:390
 [16] #invokelatest#1 at ./essentials.jl:790 [inlined]
 [17] invokelatest at ./essentials.jl:789 [inlined]
 [18] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:374
 [19] exec_options(::Base.JLOptions) at ./client.jl:312
 [20] _start() at ./client.jl:464

Am I incorrectly using the package manager or is this a dependency resolution mistake in Julia?

jling · October 13, 2019, 1:48am

I think you shouldn’t dev after add? In any case, remove any Zygote related stuff and just do ] dev Zygote should always work.

rdyro · October 13, 2019, 1:51am

That doesn’t seem to change the behavior.

% ~ % rm -rf ~/.julia
% ~ % julia --banner=no
(v1.2) pkg> dev Zygote
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `~/.julia/registries/General`
   Cloning git-repo `https://github.com/FluxML/Zygote.jl.git`
  Updating git-repo `https://github.com/FluxML/Zygote.jl.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

jling · October 13, 2019, 1:54am

I see, that means there’s a bug in current Zygote’s Dependency with ZygoteRules.

rdyro · October 13, 2019, 5:37am

The dependencies list is correct, I’m pretty sure. If I manually go through all the packages specified in the Manifest.toml, I can successfully add/install the Zygote package. See the script attached.

using Pkg

env = length(ARGS) > 0 ? ARGS[1] : "."
fname = length(ARGS) > 1 ? ARGS[2] : "Manifest.toml"

Pkg.activate(env)
deps = Pkg.TOML.parse(read(fname, String))

for key in keys(deps)
  dep = deps[key][1]

  spec = Dict{Symbol, String}()
  key_symb_map = Dict("repo-url" => "url", "repo-rev" => "rev")
  allowed_keys = ["name", "repo-url", "repo-rev", "version", "uuid", "path"]
  if haskey(dep, "repo-url") || haskey(dep, "path")
    allowed_keys = allowed_keys[allowed_keys .!= "version"]
  end
  for s in allowed_keys
    if haskey(dep, s)
      spec[Symbol(haskey(key_symb_map, s) ? key_symb_map[s] : s)] = dep[s]
    end
  end
  Pkg.add(PackageSpec(; spec...))
end

00vareladavid · October 13, 2019, 1:35pm

You have to add ZygoteRules#master before you dev Zygote.

Dependency resolution only takes registered versions into account.

rdyro · October 13, 2019, 8:15pm

The fact that Pkg resolves dependencies of a development package using the general registry seems counter intuitive.

What if I’m developing two development packages A and B at the same time and A depends on B? Pkg will throw an error because I haven’t placed B in the general registry. The Manifest.toml of A is right there and specifies exactly where B is, yet it is being ignored.

My impression is that for development packages, especially ones specified via a git link and not from the general registry, either:

Pkg should default to using sources specified in the Manifest
there should be a Pkg option to explicitly use Manifest sources

If

Dependency resolution only takes registered versions into account.

then is my script technically the solution? Isn’t there a better of way of resolving dependencies of a development package?

fredrikekre · October 13, 2019, 9:19pm

See Travis CI script does not clone Project.toml of dependent packages - #4 by fredrikekre

rdyro · October 14, 2019, 4:31am

I see, so it is the case that for now the Manifest of the dependency is ignored and you have to add them manually.

I’m linking here the pull request which fredrikekre mentions in the Travis post.

https://github.com/JuliaLang/Pkg.jl/pull/1088

Thanks.

Topic		Replies	Views
Cannot install Zygote to Julia 1.5.0 New to Julia question	1	564	September 8, 2020
Can't add Zygote in Julia 1.3 General Usage question	8	1183	December 5, 2019
Unclear where compatibility requirements are coming from New to Julia package-manager , compatibility	7	360	May 23, 2023
Package version conflicts General Usage	6	381	January 2, 2023
Compatibility problem when installing julia packages General Usage	5	1460	June 16, 2020

Seemingly incorrect dependencies resolution in Pkg

Related topics