PkgBenchmark on project with unregistered dependency

I’m struggling to get PkgBenchmark working for packages that require an unregistered dependency.

A simple demo of this issue is Example.jl, which depends on the newfeature branch of ExampleDep.jl.

Example.jl has no issues being checked-out and tested.

git clone https://github.com/milesfrain/Example.git
cd Example/
julia --project=. -e 'using Pkg; Pkg.test()'
...
   Testing Example tests passed

But attempts with PkgBenchmark fail to find ExampleDep.jl.

julia -e 'using PkgBenchmark; benchmarkpkg(".")'
...
PkgBenchmark: Running benchmarks...
Activating new environment at `/tmp/jl_BNuh4Z/Project.toml`
 Resolving package versions...
ERROR: LoadError: Unsatisfiable requirements detected for package ExampleDep [68615898]:
 ExampleDep [68615898] log:
 ├─ExampleDep [68615898] has no known versions!
 └─restricted to versions * by Example [d4a85bec] — no versions left
   └─Example [d4a85bec] log:
     ├─possible versions are: 0.1.0 or uninstalled
     └─Example [d4a85bec] is fixed to version 0.1.0

Is the this same problem covered by Pkg#492? What are the possible workarounds?

The easiest solution is to check in benchmark/Manifest.toml. For an example, see: https://github.com/tkf/ChainCutters.jl (which depends on an unregistered package https://github.com/tkf/BroadcastableStructs.jl)

Wonderful. Thanks!
I got it working now.
For reference, here’s a branch with the required fixes for this minimal example.

Having another strange issue now.

When I try to setup Project.toml and Manifest.toml in my main project, Pkg3 wants to update my simplified Example project (deleted from disk), which has no relation. Do I need to reset the registry?

(v1.2) pkg> activate .
Activating new environment at `~/projects/MainProject/benchmark/Project.toml`

(benchmark) pkg> status
    Status `~/projects/MainProject/benchmark/Project.toml`
  (empty environment)

(benchmark) pkg> add ..
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `/home/miles/temp/depdemo/Example`
ERROR: failed to fetch from /home/miles/temp/depdemo/Example, error: GitError(Code:ERROR, Class:Net, unsupported URL protocol)

Update:

Turns out that deleting ~/.julia/registries/General followed by a pkg update is not enough. There are remnants of this example project still in ~/.julia that break pkg commands in other completely unrelated and freshly-generated projects. Seems like this is one of those situations where it’s faster to just declare bankruptcy and delete ~./julia.

clones/430150191978257116/FETCH_HEAD
1:624af93193f0c073255ca746a2567d72d5a1fa35      not-for-merge   branch 'fixed' of /home/miles/temp/depdemo/Example
2:624af93193f0c073255ca746a2567d72d5a1fa35      not-for-merge   branch 'master' of /home/miles/temp/depdemo/Example
3:f62aaded21271266f784212bcc7fa3b5e84bdd0c      not-for-merge   'refs/remotes/origin/HEAD' of /home/miles/temp/depdemo/Example
4:624af93193f0c073255ca746a2567d72d5a1fa35      not-for-merge   'refs/remotes/origin/fixed' of /home/miles/temp/depdemo/Example
5:f62aaded21271266f784212bcc7fa3b5e84bdd0c      not-for-merge   'refs/remotes/origin/master' of /home/miles/temp/depdemo/Example

clones/430150191978257116/config
6:      url = /home/miles/temp/depdemo/Example