Seemingly incorrect dependencies resolution in Pkg

#1

I’m trying to develop Zygote, but can’t successfully install the latest development version from Github. When I try installing the package from the Github repo directly or trying to use the dev command with it, I get the following error:

julia> versioninfo()
Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

(v1.2) pkg> add Zygote
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Zygote ─ v0.3.4
  Updating `~/.julia/environments/v1.2/Project.toml`
  [e88e6eb3] + Zygote v0.3.4
  Updating `~/.julia/environments/v1.2/Manifest.toml`
 [no changes]

(v1.2) pkg> dev Zygote
  Updating git-repo `https://github.com/FluxML/Zygote.jl.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

shell> git clone https://github.com/FluxML/Zygote.jl.git
Cloning into 'Zygote.jl'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 5709 (delta 7), reused 15 (delta 0), pack-reused 5681
Receiving objects: 100% (5709/5709), 1012.82 KiB | 4.25 MiB/s, done.
Resolving deltas: 100% (3810/3810), done.

(v1.2) pkg> add ./Zygote.jl
   Cloning git-repo `./Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

(v1.2) pkg>

ZygoteRules is available on Github in version 0.2.0, but in the general registry only up to 0.1.0. Zygote’s Github manifest correctly specifies that ZygoteRules should be taken from a Github link and not the general registry, but the package manager does not seem to notice.

Now, maybe this isn’t the right way to go, but I tried using activate, to make sure Zygote’s manifest is used as is, but this is the result.

shell> git clone https://github.com/FluxML/Zygote.jl.git
Cloning into 'Zygote.jl'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 5709 (delta 7), reused 15 (delta 0), pack-reused 5681
Receiving objects: 100% (5709/5709), 1012.82 KiB | 3.99 MiB/s, done.
Resolving deltas: 100% (3810/3810), done.

(v1.2) pkg> add Zygote.jl/
   Cloning git-repo `Zygote.jl/`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
 Resolving package versions...
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `~/.julia/registries/General`
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4

(v1.2) pkg> activate Zygote.jl/
Activating environment at `~/Dropbox/projects/***/Zygote.jl/Project.toml`

(Zygote) pkg> add Zygote.jl/
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
  Updating git-repo `/Users/****/Dropbox/projects/****/Zygote.jl`
ERROR: UndefVarError: mode not defined
Stacktrace:
 [1] #add#25(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:96
 [2] add at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:69 [inlined]
 [3] do_add!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:496
 [4] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:790
 [5] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:789
 [6] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:403
 [7] #do_cmd#23(::Bool, ::typeof(Pkg.REPLMode.do_cmd), ::REPL.LineEditREPL, ::String) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:382
 [8] do_cmd at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:378 [inlined]
 [9] (::getfield(Pkg.REPLMode, Symbol("##28#31")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Pkg/src/REPLMode.jl:610
 [10] #invokelatest#1 at ./essentials.jl:790 [inlined]
 [11] invokelatest at ./essentials.jl:789 [inlined]
 [12] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/LineEdit.jl:2306
 [13] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:1038
 [14] run_repl(::REPL.AbstractREPL, ::Any) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:201
 [15] (::getfield(Base, Symbol("##737#739")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:390
 [16] #invokelatest#1 at ./essentials.jl:790 [inlined]
 [17] invokelatest at ./essentials.jl:789 [inlined]
 [18] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:374
 [19] exec_options(::Base.JLOptions) at ./client.jl:312
 [20] _start() at ./client.jl:464

Am I incorrectly using the package manager or is this a dependency resolution mistake in Julia?

#2

I think you shouldn’t dev after add? In any case, remove any Zygote related stuff and just do ] dev Zygote should always work.

#3

That doesn’t seem to change the behavior.

% ~ % rm -rf ~/.julia
% ~ % julia --banner=no
(v1.2) pkg> dev Zygote
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `~/.julia/registries/General`
   Cloning git-repo `https://github.com/FluxML/Zygote.jl.git`
  Updating git-repo `https://github.com/FluxML/Zygote.jl.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ZygoteRules [700de1a5]:
 ZygoteRules [700de1a5] log:
 ├─possible versions are: 0.1.0 or uninstalled
 └─restricted to versions 0.2.0-0.2 by Zygote [e88e6eb3] — no versions left
   └─Zygote [e88e6eb3] log:
     ├─possible versions are: 0.3.4 or uninstalled
     └─Zygote [e88e6eb3] is fixed to version 0.3.4
#4

I see, that means there’s a bug in current Zygote’s Dependency with ZygoteRules.

#5

The dependencies list is correct, I’m pretty sure. If I manually go through all the packages specified in the Manifest.toml, I can successfully add/install the Zygote package. See the script attached.

using Pkg

env = length(ARGS) > 0 ? ARGS[1] : "."
fname = length(ARGS) > 1 ? ARGS[2] : "Manifest.toml"

Pkg.activate(env)
deps = Pkg.TOML.parse(read(fname, String))

for key in keys(deps)
  dep = deps[key][1]

  spec = Dict{Symbol, String}()
  key_symb_map = Dict("repo-url" => "url", "repo-rev" => "rev")
  allowed_keys = ["name", "repo-url", "repo-rev", "version", "uuid", "path"]
  if haskey(dep, "repo-url") || haskey(dep, "path")
    allowed_keys = allowed_keys[allowed_keys .!= "version"]
  end
  for s in allowed_keys
    if haskey(dep, s)
      spec[Symbol(haskey(key_symb_map, s) ? key_symb_map[s] : s)] = dep[s]
    end
  end
  Pkg.add(PackageSpec(; spec...))
end
#6

You have to add ZygoteRules#master before you dev Zygote.

Dependency resolution only takes registered versions into account.

#7

The fact that Pkg resolves dependencies of a development package using the general registry seems counter intuitive.

What if I’m developing two development packages A and B at the same time and A depends on B? Pkg will throw an error because I haven’t placed B in the general registry. The Manifest.toml of A is right there and specifies exactly where B is, yet it is being ignored.

My impression is that for development packages, especially ones specified via a git link and not from the general registry, either:

  • Pkg should default to using sources specified in the Manifest
  • there should be a Pkg option to explicitly use Manifest sources

If

Dependency resolution only takes registered versions into account.

then is my script technically the solution? Isn’t there a better of way of resolving dependencies of a development package?

#8

See Travis CI script does not clone Project.toml of dependent packages

1 Like
#9

I see, so it is the case that for now the Manifest of the dependency is ignored and you have to add them manually.

I’m linking here the pull request which fredrikekre mentions in the Travis post.

Thanks.