Problems after switch to Pkg3

ScottPJones · May 25, 2018, 12:24pm

I’ve been running into a strange problem ever since the switch on master to Pkg3 building one of my packages (which works fine still on v0.6.2, and older versions of v0.7).
For some reason, a Char is being passed to the save function in:
[1] #find_registered!#37(::Bool, ::Function, ::Pkg.Types.EnvCache, ::Array{String,1}, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:1057.

Any ideas what is going wrong, and how to fix or work around this?

# Lookup package names & uuids in a single pass through registries
function find_registered!(env::EnvCache,
    names::Vector{String},
    uuids::Vector{UUID}=UUID[];
    force::Bool=false,
)::Nothing
    # only look if there's something new to see (or force == true)
    names = filter(name -> !haskey(env.uuids, name), names)
    uuids = filter(uuid -> !haskey(env.paths, uuid), uuids)
    !force && isempty(names) && isempty(uuids) && return

    # since we're looking anyway, look for everything
    save(name::String) =
        name in names || haskey(env.uuids, name) || push!(names, name)
    save(uuid::UUID) =
        uuid in uuids || haskey(env.paths, uuid) || push!(uuids, uuid)

    # lookup any dependency in the project file
    for (name, uuid) in env.project["deps"]
        save(name); save(UUID(uuid)) # THIS IS LINE 1057
    end

$ julia -e 'versioninfo()'
WARNING: Base.versioninfo is deprecated: it has been moved to the standard library package `InteractiveUtils`.
Add `using InteractiveUtils` to your imports.
 in module Main
Julia Version 0.7.0-DEV.5187
Commit 13b2f2be52 (2018-05-23 21:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD EPYC 7401P 24-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, znver1)
Environment:
  TRAVIS_JULIA_VERSION = nightly
0.00s$ if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
The command "if [[ -a .git/shallow ]]; then git fetch --unshallow; fi" exited with 0.
29.73s$ julia -e 'if VERSION < v"0.7.0-DEV" ; Pkg.add("Compat") ; Pkg.clone(pwd()) ; else ; using Pkg ; Pkg.add("Compat") ; Pkg.add(pwd()) ; end ; Pkg.build("Format"); Pkg.test("Format"; coverage=true)'
   Cloning default registries into /home/travis/.julia/registries
   Cloning registry Uncurated from "https://github.com/JuliaRegistries/Uncurated.git"
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
ERROR: MethodError: no method matching (::getfield(Pkg.Types, Symbol("#save#45")){Pkg.Types.EnvCache})(::Char)
Closest candidates are:
  save(!Matched::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:1050
  save(!Matched::Base.UUID) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:1052
Stacktrace:
 [1] #find_registered!#37(::Bool, ::Function, ::Pkg.Types.EnvCache, ::Array{String,1}, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:1057
 [2] find_registered! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:1045 [inlined]
 [3] registry_resolve!(::Pkg.Types.EnvCache, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:934
 [4] #add_or_develop#8(::Symbol, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:35
 [5] #add_or_develop at ./<missing>:0 [inlined]
 [6] #add_or_develop#7 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:22 [inlined]
 [7] #add_or_develop at ./<missing>:0 [inlined]
 [8] #add_or_develop#6 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:21 [inlined]
 [9] #add_or_develop at ./<missing>:0 [inlined]
 [10] #add_or_develop#5 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:20 [inlined]
 [11] #add_or_develop at ./<missing>:0 [inlined]
 [12] #add#9 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:43 [inlined]
 [13] add(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:43
 [14] top-level scope at none:1

mauro3 · May 25, 2018, 12:28pm

Have you tried deleting ~/.julia/registries? That worked for me, although on a different error.

kristoffer.carlsson · May 25, 2018, 12:31pm

Looks like this could happen if env.project["deps"] is a string. Have you some custom Project.toml file?

Edit: perhaps not a string but something seems weird with the project file.

ScottPJones · May 25, 2018, 12:31pm

Yes, I saw that it had worked for you, and tried that last night, but no go.
It happens both on my local machine and on Travis-CI (where it always starts from a newly downloaded registry)

ScottPJones · May 25, 2018, 12:35pm

These are the deps values for all of the packages:

./StrAPI.jl/Project.toml:5:deps = ["ModuleInterfaceTools", "Random"]
./PCRE2.jl/Project.toml:5:deps = ["BinaryProvider", "Libdl", "Test"]
./StrBase.jl/Project.toml:6:deps = ["MurmurHash3", "ChrBase"]
./Unicode_Entities.jl/Project.toml:6:deps = ["Pkg", "StrTables", "Test", "Unicode"]
./HTML_Entities.jl/Project.toml:7:deps = ["Pkg", "StrTables", "Test"]
./StrTables.jl/Project.toml:5:deps = ["Dates", "Pkg", "Test"]
./Format.jl/Project.toml:6:deps = ["Compat"]
./ModuleInterfaceTools.jl/Project.toml:5:deps = ["Test"]
./LaTeX_Entities.jl/Project.toml:6:deps = ["LightXML", "Pkg", "StrTables", "Test"]
./Emoji_Entities.jl/Project.toml:6:deps = ["JSON", "Pkg", "StrTables", "Test"]

Here is the Project.toml file:

desc = "API for Str and Chr types"
keywords = ["API", "Strings", "Characters"]
license = "MIT"
name = "StrAPI"
deps = ["ModuleInterfaceTools", "Random"]
git-tree-sha1 = "ee8eeb3b8a3bbff685c38c4815fb3b8b4c3538ee"
repo-rev = ""
repo-url = "/s/StrAPI.jl"
uuid = "7309dffa-5f6c-11e8-0bac-7b275211ca99"
version = "0.0.0"

I created it after Pkg was complaining about needing a Project.toml file.
Do you see anything obviously wrong with it?

Is there a specification documented somewhere for all of the *.toml files?
I tried to look up the documentation yesterday, but got a 404 error trying to follow the link

kristoffer.carlsson · May 25, 2018, 12:40pm

You can look at the docs in julia Base now for the Pkg standard library.

The deps section is a mapping between names and uuids. You can look at the https://github.com/JuliaLang/Pkg.jl/blob/master/Project.toml for an example.

And yes, we are painfully aware that the docs are not fleshed out yet and that there are some festures missing e.g. OS specific deps and test deps. They are being worked on.

I will open an issue for a Project.toml verifier that would detect bad project files.

kristoffer.carlsson · May 25, 2018, 12:49pm

https://docs.julialang.org/en/latest/manual/code-loading.html

is also good reading.

kristoffer.carlsson · May 25, 2018, 1:09pm

I should also say that we are working g on something that would automatically add the proper project files to a package based on the REQUIRE file. So it might be worth having a bit of patience here.

ScottPJones · May 25, 2018, 2:44pm

Thanks very much for your responses (and for your work along with Stefan in getting Pkg3 ready)!

Right, that’s where I looked yesterday, but got the 404 error (some mixup with Pkg->OldPkg, Pkg3->Pkg I assume), but it’s all good today.

I wish though that replacing Pkg with Pkg3 had been delayed until that was done, I fear it will affect a lot of people who’d like to get their packages ready for v0.7-alpha and v1.0, as time is running short

Any idea when that might be ready, or even a manual tool to check a package, and product a Project.toml file for it?

I’m running into this problem now on another package:

(MurmurHash3) pkg> add /s/MurmurHash3.jl
┌ Info: Pkg is running without precompile statements, first action will be slow.
│ Rebuild julia with the environment variable `JULIA_PKG3_PRECOMPILE` set to enable precompilation of Pkg.
└ This message can be disabled by setting the env variable `JULIA_PKG3_DISABLE_PRECOMPILE_WARNING`.
  Updating git-repo `/s/MurmurHash3.jl`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
ERROR: Cannot add package with the same name or uuid as the project

The Project.toml file is:

desc = "MurmurHash3 implementation in pure Julia"
keywords = ["Strings", "Hashing"]
license = "MIT"
name = "MurmurHash3"
repo = "https://github.com/JuliaString/MurmurHash3.jl.git"
uuid = "f0d41675-b40b-4ac0-9c24-1938651d8492"
version = "0.1.0"

What would have caused that error?

Also this error just popped up:

(Format) pkg> status
Project Format v0.6.2
ERROR: MethodError: no method matching in_project(::Array{String,1})
Closest candidates are:
  in_project(::Dict) at /j/julia/usr/share/julia/stdlib/v0.7/Pkg/src/Display.jl:255
Stacktrace:
 [1] status(::Pkg.Types.Context, ::Pkg.Types.PackageMode, ::Bool) at /j/julia/usr/share/julia/stdlib/v0.7/Pkg/src/Display.jl:51
 [2] status at /j/julia/usr/share/julia/stdlib/v0.7/Pkg/src/Display.jl:31 [inlined]
 [3] do_status!(::Pkg.Types.Context, ::Array{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Command, Pkg.REPLMode.Option, Pkg.REPLMode.Rev},1}) at /j/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:687

Thanks again, your efforts are greatly appreciated!

kristoffer.carlsson · May 25, 2018, 3:04pm

Search and replace Pkg with OldPkg, add an import OldPkg and things work just like before.

Yes, you cannot add a dependency to your named project (package) that has the same name as the project. If you just want to test the package, just do Pkg.test().

Again, this looks like a faulty Project file.

ScottPJones · May 25, 2018, 3:11pm

I could maybe do that locally, however Travis is using Pkg3 now, that might still be an issue, and also,
a big part of getting ready for v0.7/v1.0 is getting packages to work correctly with the new stuff, which is
what I’m trying to do.

As far as I was aware, I hadn’t. How might that be happening? I just have packages and the default project v0.7.

Nothing jumped out to you as obviously wrong?
I think this is an area that will need some more handling, to give clear error messages as to what is actually faulty in the Project file. (I know, be patient!)

Thanks again for the help!

kristoffer.carlsson · May 25, 2018, 3:16pm

You wrote that you did (MurmurHash3) pkg> add /s/MurmurHash3.jl.
The (MurmurHash3) pkg> says that you are in a project called “MurmurHash3” and you are adding a package called “MurmurHash3”.

The Project you posted was for MurmurHash but the error you get is from Format.
You probably still have deps = ["A", "B"] (a Vector) instead of a Dictionary ([deps] A = "uuid1...", B = "uuid2...").

ScottPJones · May 25, 2018, 8:31pm

Ah! That explains it.
I was just trying to use the Pkg REPL (nice!) to do the same as Pkg.clone("/s/MurmurHash3.jl"), which in Pkg3 has changed to Pkg.add(“/s/MurmurHash3.jl”)(I had to update all my Travis yml files to deal with that, to avoid constant warnings, both that it neededusing Pkgand thatPkg.clonewas deprecated). SincePkg.add(dirnam), I thought that add ` at the Pkg REPL would do the same.
Might need some extra documentation on that point.

I’ve only seen the syntax deps = ["A", "B"] before, in one of the example project files.
I’ll comment those lines out, see what happens.

ScottPJones · May 25, 2018, 9:27pm

OK, I’m a lot closer now to getting things all working!
The problems were because the old way of doing things when starting from scratch, like on Travis, or with an empty .julia folder, of calling Pkg.add for the registered packages, and Pkg.clone for the unregistered ones, doesn’t work at all after the Pkg3 change.
Also, it looks like some of the machines (for MacOS) are running a version before the change, so that my efforts to make things work with Pkg3 just made those fail.

Also, you can’t rely on Travis’ default test script anymore. It calls Pkg.clone(pwd()); Pkg.build(<name of package>) ; Pkg.test(...), but that’s broken (that should be a high priority to fix, I’d think).

Making a custom test script, that simply does the old way for VERSION < v0.7.0-DEV (I need to find out the exact build number to handle the Mac’s still running software from 10 days ago), and just does using Pkg ; Pkg.test(...) otherwise, seems to make things work.

Thanks for the help, the comment about just calling Pkg.test got me to figure the problem out finally!

ScottPJones · May 29, 2018, 2:38pm

Somewhat further along, however I’ve been running into another problem, which (yeah!!!) it looks like you’ve already been made aware of and fixed in #27304.
Do you have any feeling whether that will be merged soon?

kristoffer.carlsson · May 29, 2018, 2:57pm

When CI passes.

ScottPJones · May 29, 2018, 3:06pm

One thing that I’m having an issue with also, is that the manifests generated by “up” can be quite large, and not from things that are needed by the package itself, but rather, things that are only used for test (which the old system handled well, with a separate REQUIRE file).

How is that supposed to be handled with the new system? I couldn’t find anything in the docs about that.
(it would be nice, if it hasn’t been added already since I last read the docs, to have a section about “how to move from new old Pkg to the new Pkg”, including dealing with things like test/REQUIRE)

Again, thanks for your help here for getting this (much wanted) new package system whipped into shape!

kristoffer.carlsson · May 29, 2018, 3:10pm

You will be able to specify test dependencies as well.

Will look something like.

[targets.test.deps]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"

in the Project file.

These dependencies will only be active when the package is tested. I have a local branch that works and does this but some small things need to be tweaked.

ScottPJones · May 29, 2018, 3:15pm

Yeah! You are my hero today!

The one that I also really wanted was also a separate one for deps (i.e. Pkg.build) (because having tons of large dependencies, just for things like downloading and processing JSON, XML, or CSV files, in order to build tables for code that then has no dependencies at all, is rather a pain)

Does your local branch have a separate list of dependencies for Pkg.build as well?
If so, that would be super-fantastic!

kristoffer.carlsson · May 29, 2018, 3:17pm

The plan is to have a targets.build similar to targets.test which would only be active during build time. The focus right now is to get targets.test to work first because the old package manager had that and we want to get to feature parity. The extension to build time dependencies would be pretty much trivial though.