Switching package registration systems soon

package-manager
#1

Edit: This is pinned globally because it’s probably good for everyone to be aware, but this only impacts you if you are a package maintainer, in which case it changes the way you have to register new versions of packages. TLDR: starting on Monday, you’ll need to use Registrator instead of attobot to register new package versions.


We’re planning to switch from registering packages using attobot against METADATA.jl to registering packages with Registrator against the General registry on Monday. In advance of that, it would be great if people could test out Registrator and see how it goes—currently it’s setup to register new package versions with a dummy fork of General, but the process is otherwise the one that we’ll be using. If you change your registry to point at the fork, then you can use this to install packages as well. It would be greatly helpful if people could try it out.

Also, there is likely to be some down time during which one cannot register new package versions during the switchover. And depending on how it goes, there may be additional down time. So please let this be advance warning that if you really need to register a new package version for some reason, do it before Monday or hold off until later next week.

39 Likes
GitError(Code:ERROR, Class:SSL, unexpected return value from ssl handshake -9806)
#2

Will this remove the need for REQUIRE in registered packages, or not yet?

2 Likes
#3

Yes

17 Likes
#4

I tried via an issue:

let’s see what happens.

1 Like
#5

How will this affect packages that maintain compatibility with Julia v0.6 and v1.1 for example? This means that support for v0.6 in packages should be dropped from now on, or will it continue to be possible to use attobot for those after this change?

#6

IIUC the plan is to turn off Attobot, which means you’d have to register new Julia 0.6-compatible versions of your package with PkgDev.jl or manual PR against METADATA.

#7

Does this require Project/Manifest files? If so, is there a way to mass generate them for the repositories? DiffEq will need quite a few.

3 Likes
#8

Also, I imagine this makes Pkg.METADATA_compatible_uuid unnecessary/irrelevant, since all UUIDs will work fine from now, is this correct?

#9

Yes.

2 Likes
#10

The last section of the readme is not clear to me. Is it possible to register private packages?

#11

Is registration now decoupled from Github versioning? Are we encouraged to create GitHub versions in sync with GeneralRegistry versions?

1 Like
#12

You should be able to just have Julia read the contents of your Require file and then generate the Pkg.add statements in your project that will build the file. It’ll be a bit fiddly with upper-bounded dependencies I guess.

#13

Will packages currently registered in METADATA automatically get registered in General or do we need to use Registrator to make a new release?

#14

General is already what’s used by Julia v1.0+

1 Like
#15

If you can point us to a description of this process, that’d be super useful.

1 Like
#16

Will METADATA be updated with new information from General?

#17

Currently, packages are registered against METADATA.jl and that is synced to General, which is how all Julia versions ≥ 0.7 get info about package versions. Newer Julia versions do not understand the old METADATA.jl format at all, they only look at and understand General. The reason that things are still registered that way is because:

  1. We did not have any tooling for registering packages against General when 1.0 was released but we still wanted it to work with the new system, not the old one.

  2. We did not want to break the existing registration process for users of older versions of Julia.

The way to handle both issues was to keep registering against METADATA.jl and replicate that information to the new registry format and use that for installing packages from newer Julia versions. Somewhere in a data center in Cambridge MA there’s a computer with a screen session under my account running the sync script that generates General from METADATA.jl.

What we are going to do now is turn of that sync script. That means that METADATA.jl and General will both continue to exist but they will become independent: registering something in METADATA.jl will not cause it to show up in General and vice versa.

Going forward:

  • We will only allow registering new versions of packages in METADATA.jl if the version supports at least one older version of Julia (≤ 0.6). In order to register such a version, you will have to use the old-school attobot / PkgDev workflow.

  • We will only allow registering new versions of packages in General if the version supports at least one newer version of Julia (≥ 1.0). In order to register such a version, you will have to use the new-school Registrator workflow.

  • You’ll note that there can be package versions that qualify in both categories—i.e. ones that support both an older ≤ 0.6 Julia version and a newer ≥ 1.0 Julia version. In such a case, you will have to register the package twice: once with the old attobot / PkgDev workflow and another time with the new Registrator workflow. Please register the same commit with the same version number in both. If you don’t, it’s not the end of the world since whatever’s in METADATA.jl will be forgotten in the sands of time relatively soon, but it just seems bad to register different things in different registries with the same name and version number.

15 Likes
#18

No, it will not—they will be completely decoupled.

#19

Yes, the current content of General will remain as-is but going forward it will no longer be automatically updated from METADATA. Instead, it will become its own separate source of truth that is updated directly vis Registrator when registering new package versions.

1 Like
#20

Yes, you are encouraged to but not required to have git tags and GitHub releases that match registered versions. Currently, I believe Registrator comments on your repo with instructions on how to do the tagging. It previously did the tagging for you but some people didn’t like the fact that this meant that Registrator needed write access to your repo.

We may introduce a dedicated “TagBot” in the future that can be activated independently of Registrator that just creates tags for registered versions. That way people who don’t want to give write access don’t have to. It also reduces the attack surface by separation of duties: the tag bot, which needs write permission should be as simple and hard to screw up as possible, while the registration bot, which is much more complex, only has read permissions to it can’t do much harm.

9 Likes