Current best practices for depencency management for registered packages

Tamas_Papp · February 2, 2019, 9:33am

Could someone please summarize the current recommended practices for REQUIRE and Project.toml for packages that intend to support Julia 1.0 and later, and are registered via attobot?

Specifically,

do we still need a REQUIRE, or is a Project.toml enough?
should the version go in the Project.toml and be updated (because mismatches now seem to be checked and rejected), or is it OK to have a Project.toml without a version? (the latter is the easiest and least error-prone).
Should the UUID be the one from Pkg.METADATA_compatible_uuid?

fredrikekre · February 2, 2019, 10:36am

Yes, no.
You don’t need a version, but if you do it needs to match the tag when registering (METADATA.jl/check-pr.jl at 9864c6b21cb003c29ebfcf4a6575a8d558206ed6 · JuliaLang/METADATA.jl · GitHub).
Yes. (https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L168-L181)

gcalderone · February 2, 2019, 1:25pm

I confess I’m still a little confused about the package management… Two release versions have passed since the adoption of Pkg3 (v0.7 and 1.0, we are now at 1.1).
Then why it is still “yes, no”?
When will it becomes “no, yes”?
…and when the time arrives, what would happen to all those registered projects which doesn’t have a Project.toml?

This is of course not to complain about possible delays, all the work you made is definitely awesome!
It’s just to understand the rationale behind switching to Pkg3 so early in Julia history.

00vareladavid · February 3, 2019, 12:22am

Overview

Source code handling has three components: the code loader, package manager, and registration system.

The code loader is built into the language. On import, the code loader will look for code in certain locations on the filesystem.

Pkg, the package manager, is a stdlib. Pkg is responsible for setting up the filesystem so that the code loader can easily find code. If a package is not installed: Pkg will determine where that package lives on the network, download/decompress the package, and finally put the code in a location where the code loader can find it.

Pkg determines the URL for a package by querying a registry. A registry is simply a database of package metadata.

Package registration

The current registry system is still based on METADATA. METADATA only understands REQUIRE files. This is why you currently need a REQUIRE file to register a package.

General is based on the registry system that will replace METADATA. Pkg understands General, but not METADATA.

A high level view of the current registration system:

REQUIRE -> METADATA -> General <- Pkg

New environments system

Project.toml represents a system of organizing code based on environments. Recall that the source code handling system consists of three loosely coupled components. An environments-based system is compelling enough that it makes sense to upgrade components as soon as possible. The code loader and package manager have already been upgraded to use this new system.

Tradeoffs

The current setup means that package consumers can enjoy the benefits of an environment-based system today. The code loader and package manager are already quite stable and have been improving based on user feedback.

The price to pay for these benefits is having to register your package with a REQUIRE file. The fact that the current registration system is cumbersome is even more reason to wait for a better designed replacement system.

Tamas_Papp · February 4, 2019, 7:59am

Note that I wasn’t complaining about anything, and I understand that things are in transition. I just needed a clarification of about the current status for practical purposes.

00vareladavid · February 4, 2019, 8:33am

Yes, I know . I’ve seen a few people who were confused by the registration process (I certainly was), so I was just taking the opportunity to provide some context.

gcalderone · February 4, 2019, 9:14am

Me neither, really! I just wanted to understand…

And now I’m even more confused:

What is a “replacement system”? Why do we need it?

AFAIK the general registry is already working, and there is an automatic procedure copying METADATA info into it. Then why can’t we submit merge requests directly on the registry (rather than on metadata) ?

If the above is not possible, why don’t we do the opposite? i.e. register on the general registry with Project.toml and automatically generate a REQUIRE file for METADATA?

Sorry for all the questions: I don’t consider myself anymore a newbie, and certainly not an expert. Still, Pkg3 is among the most difficult concept I encountered in Julia.

And what’s more problematic is that new users face Pkg3, and its obscure error messages, way before advanced topics such as parametric types, generated functions, traits, etc… How can we convince people using Julia if even advanced beginners (such as me) don’t understand the basics of registering a package?

Please understand this not as a complain, but as a willing to understand…

Tamas_Papp · February 4, 2019, 9:29am

But having read this topic, now you do, don’t you?

I am under the impression that features are in development, and the only way to speed it up is contributing. I am confident that we will get something very nice in the end, and so I am fine with waiting. Workarounds in the meantime are OK.

00vareladavid · February 4, 2019, 9:47am

It seems I need to work on my tone . I understand no one was complaining, I was just trying to explain the reasoning for the current state of things.

What is a “replacement system” ?

why can’t we submit merge requests directly on the registry (rather than on metadata) ?

The replacement system is exactly the system that will allow us to register directly to General.

You might ask yourself why no one just tweaks attobot minimally to get the job done. Presumably this is because the whole ecosystem will depend on the new registration infrastructure once it is up. It is difficult to change behavior once so many people depend on it. It makes sense that it is not being rushed.

I’m curious, do you find working with Pkg confusing? Or is just the registration system which is not clear?

gcalderone · February 4, 2019, 11:06am

This is exactly what I wonder about. I understand it may not be straightforward, and operability of the ecosystem has a higher priority. But the very fact that we switched to Pkg3 lead me to think, maybe mistakenly, that such piece should already be in place.

Well, to create a package, according to the manual, you should run through the generate, activate, add commands to properly prepare the Project.toml and Manifest.toml files. But to register it you should throw away those files and prepare a REQUIRE file instead. Am I the only one finding this unintuitive?

Also, with Pkg3 it is possible to declare a dependency on an unregistered package, but AFAIK there is no way to declare such dependency in REQUIRE. So I can’t deploy a registered package depending on an unregistered one.

As already noted no one here is complaining, and i believe this discussion already clarified the underlying problem, namely that we miss a “replacement system”, i.e. a

My very humble opinion and proposal, is that these facts, and the natural consequence that we can not take full advantage of all Pkg3 functionalities for registered packages, should be clearly stated somewhere in an official documentation, rather than being scattered in several posts on the Discourse.

Besides, if there was such entry in an official documentation, @Tamas_Papp would not have started this discussion, and I would have been aware of the missing “replacement system” problem.

Sorry for cluttering this discussion with questions due to my ignorance, hopefully this would be useful to someone else.

fredrikekre · February 4, 2019, 11:17am

It’s actually not possible.

Tamas_Papp · February 4, 2019, 11:18am

Not at all. See eg this issue; it is unfortunate that it was closed without a solution.

Also, you may be interested in

github.com/JuliaLang/Pkg.jl

Package registration process

opened 03:56PM - 22 Oct 18 UTC

closed 09:33PM - 21 May 19 UTC

StefanKarpinski

Before we can move from METADATA being the source of truth for registered packag…es to JuliaRegistries/General being the source of truth, we need a process and infrastructure for registering packages. Here's an outline of some of my thoughts on what the process should look like. ### 1. Request **Who: package maintainer** Package maintainer proposes a new version tag - perhaps via an API endpoint? Possible data to include: - uuid - repo url - branch - tree hash - commit hash - version number Or maybe just one of `patch`, `minor` or `major` and version number is automatic based on existing version numbers. This is almost certainly too much data but I wanted to mention all the possible info one might want. Ideally we'd like to automate as much of this as possible from the repo. ### 2. Check **Who: automated** Automated validation of whether a proposed version tag is acceptable. Check that: - version number is sensible (next patch, minor or major) - install the package version by itself - run the tests for the installed package - test variations on dependencies - check if compat claims are plausible - reverse dependency testing - check semver compliance? Produces report of results ### 3. Review **Who:** - **package maintainer** (always) - **registry manager** (sometimes) Maintainer review: - accept or reject based on report If maintainer accepts, go to registry manager review: - if meets auto-approval criteria, it is fully approved without manual review - otherwise proposer can request manual review from a registry manager - abilities needed: - maintainer to give reasoning - manager to give feedback - manager to make a decision - probably good to take place on a PR somewhere ### 4. Tagging **Who:** - ideally **automated** via GitHub API - otherwise manually done by **package maintainer** Propagate git tag to package repo - should used signed tags signed by some authorized entity Can we use the github API to create tags automatically? - https://developer.github.com/v3/git/tags/#create-a-tag-object - perhaps an app that only requires that permission What would the workflow be for non-github packages? - even if it’s very manual, we should have one - have a git repo endpoint that we create tags at - then document how to pull tags from it? The tagged tree also needs to be publicly accessible but I think tagging guarantees that in git. If the tag does not match the version approved in the review step above then the version is not properly tagged and the rest of the process is blocked until the tag is fixed. ### 5. Registration **Who: automated** Once a new version has been approved and tagged, it can finally be registered. This consists of making the appropriate updates to the registry repo. This will be completely automatic.

My impression is that the current setup is considered a collection of temporary measures, so no one feels like documenting it. The best source is this discussion, but it hard to find for newbies. I asked my questions because I noticed some PRs in the related code checking the Project.toml, and I was curious.

gcalderone · February 4, 2019, 11:23am

Then I don’t understand this:

Tamas_Papp · February 4, 2019, 11:26am

You can add unregistered packages locally. It just won’t work via the registry, ie you can’t do this reliably in a registered package.

See
https://github.com/JuliaLang/Pkg.jl/issues/810

gcalderone · February 4, 2019, 11:36am

I thought adding an unregistered package would add something to Manifest.toml, which may then be depoyed with the package…

Thanks for pointing out the relevant issue!