Pkg(3): develop package which depends on local/unregistered package

Internals & Design
1

I have two unregistered packages, one depends on the other. Both have REQUIRE, neither have a Project.toml. I thought the workflow was to add them with develop, but

(v0.7) pkg> develop ~/src/julia-local-packages/TransformVariables
┌ Warning: packages will need to have a [Julia]Project.toml file in the future
└ @ Pkg.Types Types.jl:578
[ Info: Assigning UUID 1f057d3e-7f98-11e8-255d-316739be82c2 to TransformVariables
 Resolving package versions...
  Updating `~/.julia/environments/v0.7/Project.toml`
  [1f057d3e] + TransformVariables v0.0.0 [`~/src/julia-local-packages/TransformVariables`]
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [1f057d3e] + TransformVariables v0.0.0 [`~/src/julia-local-packages/TransformVariables`]

(v0.7) pkg> develop ~/src/julia-local-packages/IndirectLikelihood
┌ Warning: packages will need to have a [Julia]Project.toml file in the future
└ @ Pkg.Types Types.jl:578
[ Info: Assigning UUID 2b9eaf52-7f98-11e8-2ec6-83273b37596d to IndirectLikelihood
 Resolving package versions...
ERROR: The following package names could not be resolved:
 * TransformVariables (1f057d3e-7f98-11e8-255d-316739be82c2 in manifest but not in project)
Please specify by known `name=uuid`.

(v0.7) pkg> VERSION
ERROR: invalid command: "VERSION"

julia> VERSION
v"0.7.0-beta.133"
4 Likes
2

That is the workflow but I accidentally left out the commit that makes this possible in a previous PR that updated Pkg… (https://github.com/JuliaLang/julia/pull/27824/commits/259840d7718bdad4ca36ef4f0b30236a37842c04).

You can use the master version of Pkg to get around this or wait until the PR linked above is merged and new nightlies are out.

3

Sorry for my ignorance, but how can I do that?

I have recently learned about using julia --project in the root of Pkg, but that appears to use an empty environment, ie pkg> status has no packages. So whatever I add there will not appear in my “usual” environment, is this correct?

1 Like
4

OK, I also tried making a Project.toml for both, then inside IndirectLikelihood, add the dependency with pkg> dev path/to/TransformVariables, but

(IndirectLikelihood) pkg> status
Project IndirectLikelihood v0.1.0
    Status `Project.toml`
      [dce04be8] + ArgCheck v0.7.0
      [ffbed154] + DocStringExtensions v0.4.5
      [f6369f11] + ForwardDiff v0.7.5
      [1914dd2f] + MacroTools v0.4.2
      [d96e819e] + Parameters v0.9.0
      [eb4f743c] + TransformVariables v0.1.0 #518d56f (/home/tamas/src/julia-local-packages/TransformVariables)
[ ... manifest truncated ...]
julia> using IndirectLikelihood
[ Info: Precompiling module IndirectLikelihood
ERROR: LoadError: ArgumentError: Package IndirectLikelihood does not have TransformationsVariables in its dependencies:
 - If you have IndirectLikelihood checked out for development and have
   added TransformationsVariables as a dependency but haven't updated your primary
   environment's manifest file, try `Pkg.resolve()`.
 - Otherwise you may need to report an issue with IndirectLikelihood.

Using latest master.

5
  • Clone Pkg.
  • Remove the uuid = field in Project.toml.
  • Then dev /path/to/Pkg.

Now, import Pkg should use the files in the repo instead of the one build into the sysimg.

6

You would need TransformationsVariables in the REQUIRE file. Do you have that?

7

Yes, I have it in REQUIRE. But I also have it in Project.toml, wouldn’t that supersede REQUIRE? Both packages have a project file.

8

Yes, if it is in the Project file then that would take precedence.

I am wondering if you are hitting https://github.com/JuliaLang/julia/issues/27901 (which is now fixed https://github.com/JuliaLang/julia/pull/27906).

9

I am not sure. Removing __precompile__() from IndirectLikelihood.jl did not fix the issue, but then rming and deving the dependency again did.

BTW, I am not sure I managed to load master of Pkg.jl from the cloned repo; when following your instructions, commenting out the uuid makes it complain about the missing uuid.

10

Ah, you might have to give it a random uuid.

Here’s one.

julia> UUIDs.uuid1()
UUID("9b1dd374-803f-11e8-3c86-b3198277cbb9")
11

And now I re-enabled __precompile__(), reverted to the Pkg included in Julia master, and everything seems to work fine. Thanks!

Just to recap, if one is willing to make the transition to project files, then the recommended workflow for two unregistered projects A and B, with B depending on A is

shell> cd /wherever/A/is
(v0.7) pkg> init # and add an UUID, dependencies, etc
(v0.7) pkg> dev /wherever/A/is
shell> cd /path/of/B
(v0.7) pkg> init # and add, etc
(v0.7) pkg> activate .
(A) pkg> dev /wherever/A/is
(A) pkg> activate # no ., global env
(v0.7) pkg> dev /path/of/B

ie dev A in both B and the global env? Or is something superfluous here?

4 Likes
12

Trying to follow these instructions as well, I have an unregistered package named JuliaTeX that depends on another unregistered package named VerTeX and after following these instructions I get this

julia> Pkg.resolve()
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package VerTeX [10c596c0]:
 VerTeX [10c596c0] log:
 ├─VerTeX [10c596c0] has no known versions!
 └─restricted to versions * by JuliaTeX [fc8764c4] — no versions left
   └─JuliaTeX [fc8764c4] log:
     ├─possible versions are: 0.0.0 or uninstalled
     └─JuliaTeX [fc8764c4] is fixed to version 0.0.0

so is it only possible to load these if they have a version tagged? this new workflow is confusing me

1 Like
13

Did you have a Project.toml for at least VerTex? That’s the only way I got this working.

14

Yes, I have it, but I’m not sure what all needs to be in it, I put the name and uuid.

15

Any way that I try it, I always end up with the error message about there being no known versions of VerTeX

Is it possibly a bug, since I have version 0.0.0 there, but it can’t be picked?

16

This is likely bugged on beta so if that is what you are using, try nightly.

17

I’m using a build from yesterday night, but I just figured out my problem. What I had to do was to manually edit the ~/.julia/environments/Manifest.toml file and change the repo to a path with the dev/JuliaTeX directory, for some reason this was not correctly set by the REPL commands.

18

The manifest should never need to be edited manually. If you could reproduce what to do that makes the Manifest not update correctly, that would be helpful.

19

I’m not sure of the exact sequence of events, but I believe it had to do with a combination of switching over from an older build of Julia and the “chicken and egg” problem of cloning an unregistered repository form github without any Project.toml file. Since the repository was previously downloaded in earlier version of Julia, it was in the Manifest as a repo reference. When I tried to dev into it, it would always fail because the version requirements are unsatisfiable, which was because the environment Manifest file was referencing the repo version and not the local stateful dev version where I had the new Project.toml.

So, I think it’s a combination of having leftovers from an earlier Julia version, as well as the chicken and egg problem of cloning an unregistered package that does not contain a Project.toml. That’s my assessment.

I think it could help future users if there was some more information about how to debug potential issues like this with the package manager, but I’m not quite sure yet what the best information for that would be yet.

20

How can I make a PR to Pkg ? Maybe like this:

Then fork Pkg on github, then add a remote to the repo created above that points to the fork on github.