Current best practices for depencency management for registered packages

question

#1

Could someone please summarize the current recommended practices for REQUIRE and Project.toml for packages that intend to support Julia 1.0 and later, and are registered via attobot?

Specifically,

  1. do we still need a REQUIRE, or is a Project.toml enough?

  2. should the version go in the Project.toml and be updated (because mismatches now seem to be checked and rejected), or is it OK to have a Project.toml without a version? (the latter is the easiest and least error-prone).

  3. Should the UUID be the one from Pkg.METADATA_compatible_uuid?


#2
  1. Yes, no.

  2. You don’t need a version, but if you do it needs to match the tag when registering (https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L182-L187).

  3. Yes. (https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L168-L181)


#3

I confess I’m still a little confused about the package management… Two release versions have passed since the adoption of Pkg3 (v0.7 and 1.0, we are now at 1.1).
Then why it is still “yes, no”?
When will it becomes “no, yes”?
…and when the time arrives, what would happen to all those registered projects which doesn’t have a Project.toml?

This is of course not to complain about possible delays, all the work you made is definitely awesome!
It’s just to understand the rationale behind switching to Pkg3 so early in Julia history.


#4

Overview

Source code handling has three components: the code loader, package manager, and registration system.

The code loader is built into the language. On import, the code loader will look for code in certain locations on the filesystem.

Pkg, the package manager, is a stdlib. Pkg is responsible for setting up the filesystem so that the code loader can easily find code. If a package is not installed: Pkg will determine where that package lives on the network, download/decompress the package, and finally put the code in a location where the code loader can find it.

Pkg determines the URL for a package by querying a registry. A registry is simply a database of package metadata.

Package registration

The current registry system is still based on METADATA. METADATA only understands REQUIRE files. This is why you currently need a REQUIRE file to register a package.

General is based on the registry system that will replace METADATA. Pkg understands General, but not METADATA.

A high level view of the current registration system:

REQUIRE -> METADATA -> General <- Pkg

New environments system

Project.toml represents a system of organizing code based on environments. Recall that the source code handling system consists of three loosely coupled components. An environments-based system is compelling enough that it makes sense to upgrade components as soon as possible. The code loader and package manager have already been upgraded to use this new system.

Tradeoffs

The current setup means that package consumers can enjoy the benefits of an environment-based system today. The code loader and package manager are already quite stable and have been improving based on user feedback.

The price to pay for these benefits is having to register your package with a REQUIRE file. The fact that the current registration system is cumbersome is even more reason to wait for a better designed replacement system.


#5

Note that I wasn’t complaining about anything, and I understand that things are in transition. I just needed a clarification of about the current status for practical purposes.


#6

Yes, I know :slightly_smiling_face:. I’ve seen a few people who were confused by the registration process (I certainly was), so I was just taking the opportunity to provide some context.


#7

Me neither, really! I just wanted to understand…

And now I’m even more confused:

What is a “replacement system”? Why do we need it?

AFAIK the general registry is already working, and there is an automatic procedure copying METADATA info into it. Then why can’t we submit merge requests directly on the registry (rather than on metadata) ?

If the above is not possible, why don’t we do the opposite? i.e. register on the general registry with Project.toml and automatically generate a REQUIRE file for METADATA?

Sorry for all the questions: I don’t consider myself anymore a newbie, and certainly not an expert. Still, Pkg3 is among the most difficult concept I encountered in Julia.

And what’s more problematic is that new users face Pkg3, and its obscure error messages, way before advanced topics such as parametric types, generated functions, traits, etc… How can we convince people using Julia if even advanced beginners (such as me) don’t understand the basics of registering a package?

Please understand this not as a complain, but as a willing to understand…


#8

But having read this topic, now you do, don’t you? :wink:

I am under the impression that features are in development, and the only way to speed it up is contributing. I am confident that we will get something very nice in the end, and so I am fine with waiting. Workarounds in the meantime are OK.


#9

It seems I need to work on my tone :laughing:. I understand no one was complaining, I was just trying to explain the reasoning for the current state of things.

What is a “replacement system” ?

why can’t we submit merge requests directly on the registry (rather than on metadata) ?

The replacement system is exactly the system that will allow us to register directly to General.

You might ask yourself why no one just tweaks attobot minimally to get the job done. Presumably this is because the whole ecosystem will depend on the new registration infrastructure once it is up. It is difficult to change behavior once so many people depend on it. It makes sense that it is not being rushed.


I’m curious, do you find working with Pkg confusing? Or is just the registration system which is not clear?


#10

This is exactly what I wonder about. I understand it may not be straightforward, and operability of the ecosystem has a higher priority. But the very fact that we switched to Pkg3 lead me to think, maybe mistakenly, that such piece should already be in place.

Well, to create a package, according to the manual, you should run through the generate, activate, add commands to properly prepare the Project.toml and Manifest.toml files. But to register it you should throw away those files and prepare a REQUIRE file instead. Am I the only one finding this unintuitive?

Also, with Pkg3 it is possible to declare a dependency on an unregistered package, but AFAIK there is no way to declare such dependency in REQUIRE. So I can’t deploy a registered package depending on an unregistered one.

As already noted no one here is complaining, and i believe this discussion already clarified the underlying problem, namely that we miss a “replacement system”, i.e. a

My very humble opinion and proposal, is that these facts, and the natural consequence that we can not take full advantage of all Pkg3 functionalities for registered packages, should be clearly stated somewhere in an official documentation, rather than being scattered in several posts on the Discourse.

Besides, if there was such entry in an official documentation, @Tamas_Papp would not have started this discussion, and I would have been aware of the missing “replacement system” problem.

Sorry for cluttering this discussion with questions due to my ignorance, hopefully this would be useful to someone else.:wink:


#11

It’s actually not possible.


#12

Not at all. See eg this issue; it is unfortunate that it was closed without a solution.

Also, you may be interested in

My impression is that the current setup is considered a collection of temporary measures, so no one feels like documenting it. The best source is this discussion, but it hard to find for newbies. I asked my questions because I noticed some PRs in the related code checking the Project.toml, and I was curious.


#13

Then I don’t understand this:


#14

You can add unregistered packages locally. It just won’t work via the registry, ie you can’t do this reliably in a registered package.

See


#15

I thought adding an unregistered package would add something to Manifest.toml, which may then be depoyed with the package…

Thanks for pointing out the relevant issue!