Some more observations on working with Pkg(3)

General Usage
1

Encouraged by this thread ( Switching to Pkg3 )
I also ventured into Pkg(3) land for a redo of Stan.jl for Julia 0.7/1.0 ( StanJulia ) and similarly for a previously registered project PtFEM.jl ( PtFEM ). Both Github organizations (StanJulia and PtFEM) currently hold several related work-in-progress packages.

And as Scott mentioned, Pkg(3) is very different, I like it though. Both StanJulia/CmdStan.jl and PtFEM/PtFEM.jl are maybe 80% there.

I’m left with several questions/observations, mainly workflow related:

  1. It seems Pkg(3) expects Julia to run in the ‘correct’ directory? E.g. after generate HelloWorld ( as in the Pkg docs ) I found the project in the current working directory. The Pkg REPL also relies on that I think.

  2. I like this idea of having several Julia processes running, each bound to a project. Is that the idea? Pretty quickly I found myself constantly working in the Pkg REPL for a particular project.

  3. (After saving the v0.6 subdirectory in .julia,) I tried if I could delete all of .julia… That did work ok, dev, packages, registries, etc are recreated when needed.

  4. All previously registered packages are in .julia/packages. These are not (never?) git repositories?

  5. The version of a package that Julia selects with e.g. using PtFEM is solely determined by what is in Project.toml and Manifest.toml?

  6. When in the Pkg REPL (or using Pkg.add(…)), will add package_name always select the most suitable registered version available? To override that, use an URL or local directory?

  7. After working ( develop ... ) on a package in e.g. .julia/dev, the package is now ready for a new release. The Pkg documentation states: “When the PR has been merged we can go over to track the master branch and when a new release …”, does that mean free package_name by default will switch to master and on a future up package_name to the new release?

  8. In my case review free PtFEM returned an error (see below). I’ve also seen error messages related to package and project name being the same. This is clearly the area I struggle with most. I guess the Pkg docs refer to existing packages added to a project. In my case both CmdStan.jl and PtFEM.jl are the targets of the development work.

  9. Still need to study switching between branches. In fact I’m hoping that after Julia 1.0 is released I can go back to just having a released version and a master branch.

  10. Also I’m not sure what the extra directory layer in .julia/packages/packagename/xxxx/ is for, I guess it is for version management using Manifest.toml?

  11. Over the last couple of years I have learned the hard way that it is better to layer package dependencies as much as possible. I haven’t studied that part of Pkg(3) yet. For now I plan to create base packages, e.g. CmdStan.jl and then layer creating DataFrames in a separate package (StanDataFrames). Similarly for StanMamba, StanPlotsRecipes, StanFeather etc., all part of the StanJulia organization. But I’ll certainly keep an eye on the discussion about classes of packages in Github organizations.

  12. The updates Scott suggested for .travis.yml (and appveyor.yml) seem to do the trick. This seems to simplify .travis.yml, e.g.: " - julia -e ‘using Pkg; Pkg.add(“…”); Pkg.test(“CmdStan”; coverage=true)’ ". Although I’m not sure why e.g. the Pkg.add(“Compat”) is required in .travis.yml and appveyor.yml.

  13. Will there be a Curated registry in the future?

  14. Is the REQUIRE file still needed?

Just wanted to capture these notes for my own use, but maybe they are helpful for others (as Scott’s thread was to me). Feedback is of course always welcome.

Rob

(PtFEM) pkg> preview free PtFEM
───── Preview mode ─────
ERROR: MethodError: no method matching get(::Nothing, ::String, ::Bool)
Closest candidates are:
get(::Base.EnvDict, ::AbstractString, ::Any) at env.jl:77
get(::REPL.Terminals.TTYTerminal, ::Any, ::Any) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/REPL/src/Terminals.jl:174
get(::IdDict{K,V}, ::Any, ::Any) where {K, V} at abstractdict.jl:624
…
Stacktrace:
[1] #free#30(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:187
[2] free at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/API.jl:172 [inlined]
[3] do_free!(::Pkg.Types.Context, ::Array{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Command, Pkg.REPLMode.Option, Pkg.REPLMode.Rev},1}) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:664
[4] #invokelatest#1 at ./essentials.jl:667 [inlined]
[5] invokelatest at ./essentials.jl:666 [inlined]
[6] do_cmd!(::Array{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Command, Pkg.REPLMode.Option, Pkg.REPLMode.Rev},1}, ::REPL.LineEditREPL) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:273
[7] #do_cmd#8(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:233
[8] do_cmd at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:230 [inlined]
[9] (::getfield(Pkg.REPLMode, Symbol(“##27#30”)){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v0.7/Pkg/src/REPLMode.jl:948
[10] top-level scope

2 Likes
2

The current environment (Project.toml + Manifest.toml) is based on the current dir, yes.

The idea is that you can multiple versions of packages installed at the same time and depending on what your current project is you use different ones of them. That way you can always come back to your old projects and everything will be exactly as you left them.

They are not, and packages here are (by convention) considered immutable. You should never change anything here.

Yes.

It will select the highest registered version that is compatible with all other dependencies of the project. If you want to use unregistered versions you can add use a branch, e.g. add Example#master or add a url. You can also dev something which means that you are using a local path for the package.

Free can remove a pin from a package (fixing its version) or make a package stop tracking a git repository and go back to just use registered versions again.

It’s for all different versions for packagename installed

It is needed right now to set constraints on what version of your dependencies you support. But support for that within Pkg3 is coming very soon.

2 Likes
3

Thanks a lot Kristoffer!

An item I did not include is the test facility in Pkg REPL. Somehow as I was making Julia 0.7/1.0 related changes to both packages (in different screens/terminals of course), I never had to restart Julia.

That is a huge step forward! ( Maybe that would have worked in the past similarly if I had done all testing using Pkg.test(package_name), but in Pkg(3) this feels really natural.

Rob

2 Likes
4

One other observation: I do have (v0.7) Project.toml and Manifest.toml files, they are empty. Is there a reason they would ever need to be populated?

5

Ah, perfect, I wondered about that myself. You are batting 1000 lately, Kristoffer!

That was just for that particular package, which needed Compat, and since it was unregistered, I had to have the custom build script to add any registered packages that it needed, and clone any other unregistered ones, to make it work with the old package manager.
I am in the process of getting all of the packages in the JuliaString org registered, but because of the interdepencies, it’s a long process.

I’ll post an updated version of what I had to do to get everything working, as soon as I get everything working :slight_smile:
(Right now, I think that may depend on Kristoffer’s fix getting merged in time to be in the nightlies that travis uses)

1 Like
6

What surprised me is that it appeared as if by just adding a Pkg.add(“…”) in there, it triggered something ( almost like a proper Pkg REPL instantiate action ) on both Travis and Appveyor. I haven’t tried adding too many packages, but e.g. for PtFEM.jl I could use at least either Pkg.add(“LinearAlgebra”) or Pkg.add(“Compat”).

Not having a Pkg.add(“…”) in there made both Travis and Appveyor fail.

Right now CmdStan.jl and PtFEM.jl seem ok (except I need to make many more Julia 0.7/1.0 changes similar to what I used for testing).

CmdStan.jl is not registered, but it seems to find the other required packages once the above Pkg.add() is there. Not sure if Travis and Appveyor use REQUIRE or Project.toml for that.

7

Since you are being so good at answering questions today :grinning:, I’ve got a few more I ran into:
What is the difference between: repo = "...", and repo-rev = "" ; repo-url = "..."

Ah, and another thing, when a new package manager was being discussed on GitHub a long while ago,
I remember that Tom Breloff had been asked for the ability to set up other “curated” registries of packages,
that would be “vetted” in some fashion (hopefully not “extreme vetting” though! :sweat_smile:), based on Julia
organizations (i.e. so JuliaMath would take care of registering the packages in that repository (and possibly even other related packages outside the org, say in personal repos, like stevengj/DecFP.jl)).

Will that be possible?

Thanks!

8
  • repo is in the registry, it is the repo url of the registered packages.
  • repo-rev is in the manifest, it says what git revision we are currently tracking, it can be either a commit or a branch.
  • repo-url is in the manifest and is the url of the package, note that this might not be a registered package

The registries are in .julia/registries. By default, the “Uncurated” registry gets cloned there but with Pkg3 (in contrast to Pkg2) it is possible to use multiple registries at the same time. Anyone can make their own registry with whatever packages they want while at the same time using the Uncurated registry. Just clone it to .julia/registries and Pkg3 will look there for packages as well. There is nothing special about the Uncurated registry except that it gets cloned by default when the registry folder does not exist.

9

My question was more about having other curated registries, which has been brought up before, possibly on a Julia organization basis, which would be cloned by default etc. when the registry folder doesn’t exist.
It would allow for a more de-centralized trust structure, and probably greatly reduce the number of packages that are in the “Uncurated” registry.

Tom had a number of good arguments in the GitHub discussion about why this would work well.
IMO, there’s simply too many packages in METADATA.jl for them to be adequately curated, but doing it the org level, and having the orgs “vetted”, would really help being able to provide well curated trustable registries of packages, without all the delays of trying to get things reviewed and registered in a single monolithic registry.

10

I’ve also just started really digging into this today. Wow, it is fancy. I think you are going to get people wanting to use Julia because of the package manager alone, I know some people who will go bananas over this. Thanks for everyone’s hard work on this, I know it can’t be a particularly fun project, but obviously a vital one.

Anyway, a couple of questions of my own, hopefully they are not repeats:

  • Is there a way to see the dependency tree of a package compactly in the REPL?
  • It seems that if I do develop PkgName and later rm it the repo remains in dev. Is there a “proper” way of deleting it, or can I safely remove it from the file system?
  • Are checkout and develop the same thing?
1 Like
11

Not directly, but the whole dependency graph is available in the Manifest.

Pkg3 will never touch anything outside .julia/packages since those might have local edits. You can do whatever you want with them.

Sort of, develop is for packages that you want to edit (or develop). If you do not want to edit the files, and you are just a user you can just add Plots#master and you will get the master branch of Plots. So add is when you want to use package, develop is when you want to develop for them.

1 Like
12

Right now, there are no plans for that in the immediate future, at least that I know of.

13

Correct me if I’m wrong, but I thought I saw that when I do checkout Plots it puts a clone of master in dev and uses that. That’s why I asked if develop and checkout were the same (and I’m assuming that add Plots#master is equivlaent to checkout Plots). Is that correct, or did my directory in dev come from somewhere else?

14

Aha, yeah, in Pkg3 they are the same but checkout got renamed to develop.

1 Like
15

Cool. Perhaps deprecate checkout? Although I do find that name a little clearer than develop since it is standard git terminology…

16

Yes, but that’s why it is misleading. You can develop a local path to a package that isn’t even a git repository. The purpose is just to put a path entry in your Manifest. That you can do dev PackageName and have it download is just a convenience.

1 Like
17

I find (in Pkg REPL) status (or status --project) and status --manifest quite handy and compact, but not as complete (or showing the tree structure) as Manifest.toml. Maybe room for a status -t?

As you said, fancy, as in fancy good.

2 Likes
18

Ah, status --manifest was basically what I was looking for.

Yeah, people are going to go nuts over this, it basically makes it maximally trivial to deploy things to containerized environments and it just feels slick. I’m getting close to having to deploy a Julia project in a “production” environment on AWS and this takes me from being behind everybody else in terms of containerization to being way ahead. :smile:

On that note, one thing that seems to be missing is some sort of command line entry point to the package manager. Right now it looks like you have to do julia -e "Pkg.add("SomePackage")" etc., it would be nice to have julia -P add SomePackage or something like that. (I didn’t see anything like this in man julia.) Any plans for how Julia prep lines might ultimately look?

19

By the way, one more question, if one does add SomePackage, is the dependency added fixed at whatever was the most recent version when the command was called? Or is it just the most recent version period unless you manually specify one? I think the latter would be better, I am a little worried about this all making it way too easy for people to keep packages stuck on old legacy code.

20

I had this PR add cmdline binary by KristofferC · Pull Request #42 · JuliaLang/Pkg.jl · GitHub but at the time the compilation time of Pkg was too slow for it to be useful. Should be easy to revive though.

1 Like