Lost with new Pkg3

General Usage
1

Hi,

I am totally lost with new Pkg3. I have my own packages in my own directory say ~/OwnPkg. In julia-0.6, LOAD_PATH contained this directory and everything worked nice. At the moment, I am creating Project.toml in every package by running generate pkgname to get uuid. But what I am lost at the moment is how add dependency of my package on other my own package. The recommended way using add in Pkg’s repl does not work. Adding the package manually together with its uuid into Project.toml works well, but the resolve command fails,

(Foo) pkg> resolve
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package MyPkg [1d0525e4]:
 MyPkg [1d0525e4] log:
 ├─MyPkg [1d0525e4] has no known versions!
 └─restricted to versions * by an explicit requirement — no versions left

where Foo depends on MyPkg.

Thanks for answers in advance. It is frustrating to some extent.

1 Like
2

Did you read the documentation?

In any case, activate the package, then add or dev your dependencies (dev if they are a package you are concurrently working on).

3

I read the documantation, but I have not figured that out.

4

Which way is recommended? In the manual there is a couple of sections about exactly this. Specifically you can add path/to/unregistered/package, add www.unregistered-package.com or dev path/to/unregistered/package, depending on what you want to achieve and/or where your package code exists.

If you just manually add the name and uuid for a package there is no way for Pkg to know where to find that package, so obviously it can not find any versions when doing the resolve. What you have to do is to either 1) use add with a path, or a url if you have your (unregistered) package on github or similar, or 2) use dev with a path to your package. I would recommend using dev here, since that will track the changes in your code.

5

And also, please suggest clarifications to the manual, it is new users like yourself that are in the best position to see what is lacking and/or not explained enough. You can make a PR to https://github.com/JuliaLang/Pkg.jl/blob/master/docs/src/index.md and become a developer while you’re at it!

6 Likes
6

@Tomas_Pevny, you can also suggest clarifications here.

7

I have chosen the latter option, as I do not really want to mingle with someone else docs.

8

It’s not super obvious from the docs how to develop multiple local packages that depend on each other in some way. Here’s a simple example for a package ~/MyPackages/Foo that depends on ~/MyPackages/Bar.

First time setup:

$ pwd
~/MyPackages
$ julia
(v1.0) pkg> generate Foo
(v1.0) pkg> generate Bar
shell> cd Foo
(v1.0) pkg> activate .
(v1.0) pkg> dev ../Bar

Edit Foo/src/Foo.jl and Bar/src/Bar.jl

module Bar
export bar

bar = x->x*x

end # module

module Foo
using Bar

foo = x->3*bar(x)

end # module

This works now:

$ pwd
~/MyPackages
$ julia
julia> push!(LOAD_PATH, ".")
julia> using Foo
julia> Foo.foo(10)
300

Add a new dependency on a local package Baz to Foo

$ pwd
~/MyPackages
$ julia
shell> cd Foo
(v1.0) pkg> activate .
(v1.0) pkg> dev ../Baz
shell> cd ..
shell> tree
.
├── Bar
│   ├── Project.toml
│   └── src
│       └── Bar.jl
├── Baz
│   ├── Project.toml
│   └── src
│       └── Baz.jl
├── Foo
│   ├── Manifest.toml
│   ├── Project.toml
│   ├── src
│   │   └── Foo.jl

work from ~/MyPackages
cd, activate, dev to add a local dependency.
push!( LOAD_PATH, “.” ) and using Foo anytime.

This seems like a simple workflow.

3 Likes
9

This works well for me:

  1. For each local package, activate path_to_package, then add path_to_dependency1 path_to_dependency2 ...
  2. In your top-level environment (eg. v0.7), dev path_to_package1, dev path_to_package2, etc.

Then work from your top-level environment. Each package will know how to find its dependencies, and you just told Pkg that in the current (top-level) environment, you want to use the development version of all your packages.

What really confused me was that (MyPackage) pkg> add Dependency1#master or dev Dependency1 both look like reasonable ways of telling Julia that MyPackage depends on Dependency1’s latest master/current code. But it’s absolutely not the right way to go about it, even if it misleadingly seems to work in the short term.

1 Like
10

Hello everybody,

thank you very much for those explanations. This is enlightening and should go into the documentation. Maybe the original contributors in this thread could add a direct link to the relevant discourse posts to #689 ? There is an (unrelated ?) comment about a similar (same) question, but no link to the discourse discussion and solution it is referring to.

11

Definitely still getting lost with Pkg3, but I think I’m getting better at figuring out how to actually get it to work.

I was going to ask for help, but in trying to describe the problem I eventually go things to work. I’ll leave those descriptions here though, to exemplify the difficulty / why I find Pkg3 to not be especially intuitive.

Trying to install an unregistered package that depends on another unregistered package:

(v1.0) pkg> dev https://github.com/BriceonWiley/ConfidenceIntervalComparisons.jl
  Updating git-repo `https://github.com/BriceonWiley/ConfidenceIntervalComparisons.jl`
ERROR: expected a `name` entry in project file at /tmp/tmpKDBgq6/Project.toml

(v1.0) pkg> dev /home/chriselrod/.julia/dev/ConfidenceIntervalComparisons.jl
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package AsymptoticPosteriors [5ebe76dc]:
 AsymptoticPosteriors [5ebe76dc] log:
 ├─AsymptoticPosteriors [5ebe76dc] has no known versions!
 └─restricted to versions * by ConfidenceIntervalComparisons [904d08c6] — no versions left
   └─ConfidenceIntervalComparisons [904d08c6] log:
     ├─possible versions are: 0.1.0 or uninstalled
     └─ConfidenceIntervalComparisons [904d08c6] is fixed to version 0.1.0

This is the Project.toml:

name = "ConfidenceIntervalComparisons"
uuid = "904d08c6-8d09-11e8-1189-87850a518ab4"
authors = ["Briceon Wiley <Briceon_Wiley@baylor.edu>"]
version = "0.1.0"

[deps]
AsymptoticPosteriors = "5ebe76dc-7dd7-11e9-2992-2706ab89a682"
Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"
Cubature = "667455a9-e2ce-5579-9412-b964f529a492"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
DifferentiableObjects = "e829a12a-7dbf-11e8-18a8-f94f525a9d85"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
SIMDArrays = "604d08c6-8d09-11e8-1189-87850a518ab4"
StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
jBLAS = "814864de-88d5-11e8-1c72-cfea32f978d7"

It clearly has a name. Most of those changes are local, but the name also exists on Github here.

When activating ConfidenceIntervalComparisons…

(ConfidenceIntervalComparisons) pkg> st
Project ConfidenceIntervalComparisons v0.1.0
    Status `Project.toml`
  [5ebe76dc] + AsymptoticPosteriors v0.1.0 [`~/.julia/dev/AsymptoticPosteriors`]

Located at ~/.julia/dev/AsymptoticPosteriors, I have

name = "AsymptoticPosteriors"
uuid = "5ebe76dc-7dd7-11e9-2992-2706ab89a682"
authors = ["chriselrod <elrodc@gmail.com>"]
version = "0.1.0"

However, I noticed st from the v1.0 environment lists a different uuid for AsymptoticPosteriors than what it actually has (even though it loads the library correctly!):

(v1.0) pkg> st
    Status `~/.julia/environments/v1.0/Project.toml`
  [7d9fca2a] Arpack v0.2.3+ [`~/.julia/dev/Arpack`]
  [c69eeb19] AsymptoticPosteriors v0.1.0 [`~/.julia/dev/AsymptoticPosteriors`]

My first instinct was to update Julia’s Project.toml to match what the package says. However, that turned out not to be a good idea:

(v1.0) pkg> resolve
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package AsymptoticPosteriors [5ebe76dc]:
 AsymptoticPosteriors [5ebe76dc] log:
 ├─AsymptoticPosteriors [5ebe76dc] has no known versions!
 └─restricted to versions * by an explicit requirement — no versions left

So, now I instead reverted that change, and replaced the instances of AsymptoticPosteriors’ uuid in the Manifest.toml and Project.toml of ConfidenceIntervalComparisons with the same uuid from Julia 1.0’s Project.toml. Now, finally, I could install that package:

(v1.0) pkg> dev /home/chriselrod/.julia/dev/ConfidenceIntervalComparisons.jl
 Resolving package versions...
  Updating `~/.julia/environments/v1.0/Project.toml`
  [904d08c6] + ConfidenceIntervalComparisons v0.1.0 [`~/.julia/dev/ConfidenceIntervalComparisons.jl`]
  Updating `~/.julia/environments/v1.0/Manifest.toml`
  [324d7699] + CategoricalArrays v0.3.13
  [944b1d66] + CodecZlib v0.5.0
  [904d08c6] + ConfidenceIntervalComparisons v0.1.0 [`~/.julia/dev/ConfidenceIntervalComparisons.jl`]
  [667455a9] + Cubature v1.2.2
  [a93c6f00] + DataFrames v0.13.1
  [9a8bc11e] + DataStreams v0.4.1
  [31c24e10] + Distributions v0.16.4
  [90014a1f] + PDMats v0.9.5
  [1fd47b50] + QuadGK v2.0.2
  [189a3867] + Reexport v0.2.0
  [4c63d2b9] + StatsFuns v0.7.0
  [3bb67fe8] + TranscodingStreams v0.8.1
  [ea10d353] + WeakRefStrings v0.5.3
  [9fa8497b] + Future 
  [4607b0f0] + SuiteSparse

Where did that other, “correct”, uuid come from? Can I find it anywhere other than the Project.toml?
For the other libraries I checked, Julia’s Project.toml uuid matched the canonical one specified by the package. Should I update AsymptoticPosteriors’s Project.toml to match Julia v1.0’s, even though things seem to work as is? (Ie, using AsymptoticPosteriors loads the latest changes I made to that library).

Why did resolve and st after activating ConfidenceIntervalComparisons work with the “wrong”/canonical uuid of AsymptoticPosteriors, but not within Julia 1.0’s repl?

12

If a unregistered package A depends on another unregistered package B you need to first add B, then A, just like the old package manager. The reason is that if you add A, then Pkg does not know where to find B, since it is not in the registry.

If both your packages are unregistered and have project files, then the correct uuid is/should be listed in their project files.

1 Like
13

Yes, AsymptoticPosteriors was already installed the first time I tried to install ConfidenceIntervalComparisons, which depends on the former.
The difficulty with finding the package came from the uuid mismatch.

Okay, I will correct the uuid in AsymptoticPosterior’s Project.toml then.

14

This sounds backward, but should be fine since they are not registered. My guess is that you added AsymptoticPosterior as a dependency to the other package before it had a project file, and Pkg thus assigned a random uuid for it. Normally Pkg would use the uuid in the project file here, so that file is the “truth” so to speak.

15

The uuids specified by AsymptoticPosteriors and Julia 1.0’s Project.toml files disagreed.
When I added the AsymptoticPosteriors dependency inside the ConfidenceIntervalComparisons environment, it took the uuid from AsymptoticPosteriors.

When trying to install so I could load it from Julia v1.0, it compared the uuid of the stated dependency with the uuid of the 1.0 Project.toml, saw the disagreement, and (being unregistered), threw an error because it couldn’t look anywhere to find it.
All the Project.toml’s agree now

I don’t know what I did to cause the mismatch in the first place.
If you install a package without a uuid, does Julia assign it one in it’s own Project.toml?
If so, I think what happened is that I did not know that, and then proceeded to randomly create one of my own when making a Project.toml. I did that a few times on different machines, so I’ll have to hunt for possible mismatches!

Thanks for the help.