"Unsatisfiable requirements" when local packages use local packages (fixed in v1.4)

General Usage
1

I have a project with a few local packages. Some packages use dev path/to/Package to use another package, but this only works for one level. That is, if activate PkgB and do dev path/to/PkgA, then if we activate PkgC and do dev path/to/PkgB, we’ll get a warning about “Unsatisfiable requirements detected for package PkgA”. The commands to recreate this are below.

My understanding is that dev’d packages aren’t part of the Manifest.toml, so when PkgC is trying to dev PkgB, it can’t figure out what/where PkgA is.

For my case, it does not make sense to make these packages separate git repos and use them as unregistered packages. The code must be local and all in one repo, as this is part of a larger project, or which Julia is only a tiny (but growing!) part.

Is there a way to make local packages “first class”? That is, we just need to tell the manifest about the local path, or something like this.

Here are those commands:

julia> import Pkg

julia> Pkg.generate("PkgA")
Generating project PkgA:
    PkgA\Project.toml
    PkgA/src/PkgA.jl
Dict{String,Base.UUID} with 1 entry:
  "PkgA" => UUID("742a3e6f-54ea-4f98-9a06-d7829e97a94c")

julia> Pkg.generate("PkgB")
Generating project PkgB:
    PkgB\Project.toml
    PkgB/src/PkgB.jl
Dict{String,Base.UUID} with 1 entry:
  "PkgB" => UUID("45f31414-b817-4f24-9a66-b38385072404")

julia> Pkg.generate("PkgC")
Generating project PkgC:
    PkgC\Project.toml
    PkgC/src/PkgC.jl
Dict{String,Base.UUID} with 1 entry:
  "PkgC" => UUID("ec7805be-d493-4525-8c88-e1eff6e8520a")

(v1.3) pkg> activate PkgB
Activating environment at `C:\Users\Tucker\Desktop\PkgB\Project.toml`

(PkgB) pkg> dev PkgA
[ Info: resolving package identifier `PkgA` as a directory at `C:\Users\Tucker\Desktop\PkgA`.
 Resolving package versions...
  Updating `C:\Users\Tucker\Desktop\PkgB\Project.toml`
  [742a3e6f] + PkgA v0.1.0 [`..\PkgA`]
  Updating `C:\Users\Tucker\Desktop\PkgB\Manifest.toml`
  [742a3e6f] + PkgA v0.1.0 [`..\PkgA`]

(PkgB) pkg> st
Project PkgB v0.1.0
    Status `C:\Users\Tucker\Desktop\PkgB\Project.toml`
  [742a3e6f] PkgA v0.1.0 [`..\PkgA`]

(PkgB) pkg> activate PkgC
Activating environment at `C:\Users\Tucker\Desktop\PkgC\Project.toml`

(PkgC) pkg> dev PkgB
[ Info: resolving package identifier `PkgB` as a directory at `C:\Users\Tucker\Desktop\PkgB`.
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PkgA [742a3e6f]:
 PkgA [742a3e6f] log:
 ├─PkgA [742a3e6f] has no known versions!
 └─restricted to versions * by PkgB [45f31414] — no versions left
   └─PkgB [45f31414] log:
     ├─possible versions are: 0.1.0 or uninstalled
     └─PkgB [45f31414] is fixed to version 0.1.0
1 Like
2

One silly work-around is to have PkgC first dev PkgA and then dev PkgB. With many little packages, this becomes very tedious.

3

dev them all at once instead.

1 Like
4

Thanks @fredrikekre. I think by this:

dev them all at once instead.

you mean dev PkgA PkgB, no? If so, that’s not really getting at the problem. The point is that PkgC now has to dev all of the local dependencies for every package it uses, which is hard to maintain. That is, if you add a new dependence to A, you have to go add it to anything that uses A, and to anything that uses anything that uses A, etc., which is clearly a big mess.

5

Yes, that’s unfortunate, we have this “problem” too. I think the solution is to either avoid having too many small packages, or to bite the bullet and make a registry. We did it recently with LocalRegistry.jl, and it wasn’t that bad, but we aren’t tagging yet, which is AFAIK required for the solution to the transitive dependency problem.

1 Like
6

This thread mentions a broader set of the problems experienced here: Could we make first-class support for packages that are "just files" and not repositories?

7

I’m not quite sure what you mean here. Git tags are useful in many ways but they play no part in Pkg dependency resolution. Only the information in the registry is used for that and the interaction with the package repositories is through tree hashes stored in the registry.

That said, support for adding tags corresponding to registered tree hashes is on my radar for LocalRegistry.

2 Likes
8

Yes, I meant “registering tree hashes”. We do not register tree hashes with regularity in our registry (except the first registered tree hash, of course). If A’s environment depends on B’s master, and B “depends” on C’s master, then adding A will add C’s latest registered tree hash, and not C’s master. I thought that was OP’s problem. So you’re stuck either adding C’s master to A’s environment, or registering tree hashes with regularity. I wouldn’t mind registering tree hashes, but IIRC it involves 1. changing the Project.toml version 2. committing. 3. Using localregistry 4. git tagging, 5. pushing the package and the registry. I always seem to bungle one step or another.

9

Hey Cédric, I think this still requires that each package is a git repo though, right?

10

Yes! Sorry, I missed that part of your original post.

11

Would it be possible for you to update to v1.4? It looks like this exact set of commands now works.

My hypothesis is that newer versions of Pkg are now able to look into B’s manifest to find where A comes from… Of course, you’ll have to commit all Manifest.toml files into your git repository, but one advantage of your current setup (all local packages in the same repo) is that relative paths stored in the manifests remain valid wherever that repository is cloned.

1 Like
12

For now, but it’s not a fundamental restriction. Pkg can already handle packages that are subdirectories of a git repository in a limited capacity, see https://github.com/JuliaLang/Pkg.jl/issues/1251. What is missing (except for Pkg improvements) is tooling to register such packages. In fact it was surprisingly easy to implement in LocalRegistry but I’ll wait with releasing it until [RFC] Packages in subdirectories need more registry information · Issue #30 · JuliaRegistries/RegistryTools.jl · GitHub is settled and implemented.

1 Like
13

I can verify that the commands in the original post work as expected in v1.4. Here’s the final step:

(PkgC) pkg> dev PkgB
[ Info: Resolving package identifier `PkgB` as a directory at `C:\Users\Tucker\PkgB`.
Path `PkgB` exists and looks like the correct package. Using existing path.
  Resolving package versions...
   Updating `C:\Users\Tucker\PkgC\Project.toml`
  [b64b871a] + PkgB v0.1.0 [`..\PkgB`]
   Updating `C:\Users\Tucker\PkgC\Manifest.toml`
  [83813236] + PkgA v0.1.0 [`..\PkgA`]
  [b64b871a] + PkgB v0.1.0 [`..\PkgB`]

For my repo, I think this is worth updating for!

1 Like