Announcement: automatic merging for the General registry

I’m pleased to announce that we are rolling out the first iteration of automatic merging for the General registry.

In order for your pull request to be automerged, it needs to meet the automerge guidelines:

New packages

  1. Normal capitalization

    • name should match r"^[A-Z]\w*[a-z][0-9]?$"
      • i.e. starts with a capital letter, ASCII alphanumerics only, ends in lowercase
  2. Not too short

    • at least five letters
      • you can register names shorter than this, but doing so requires someone to approve
  3. Standard initial version number

    • one of 0.0.1, 0.1.0, 1.0.0
  4. Repo URL ends with /$name.jl.git where name is the package name

  5. Compat for all dependencies

    • all [deps] should also have [compat] entries (and Julia itself)
    • all [compat] entries should have upper bounds
  6. Package can be installed

    • given the proposed changes to the registry, can we resolve and install the package?
  7. Package can be loaded

    • once it’s been installed, can we load the code?

New versions of existing packages

  1. Sequential version number

    • if the last version was 1.2.3 then the next can be 1.2.4, 1.3.0 or 2.0.0
  2. Compat for all dependencies

    • all [deps] should also have [compat] entries (and Julia itself)
    • all [compat] entries should have upper bounds
  3. Version can be installed

    • given the proposed changes to the registry, can we resolve and install the new version of the package?
  4. Version can be loaded

    • once it’s been installed, can we load the code?

Important note

These guidelines are intended not as requirements for packages but as very conservative guidelines, which, if your new package or new version of a package meets them, it may be automatically merged.

46 Likes

Also, please note that commenting on a pull request will automatically disable automerging on that pull request. Therefore, if you want to leave a comment on a pull request but you still want that pull request to be automerged, please include the text [noblock] in your comment.

2 Likes

Can you either give an example of how to write the compat statements or point to it somewhere in the docs? I’m very interested in having this autonomy!

https://julialang.github.io/Pkg.jl/v1/compatibility/

2 Likes

Cool thank you!

1 Like

Great, for registrations currently in the backlog, should we bump them to trigger zeptodoctor or is it best to just wait for a kind human registrator to merge them?

Ps: compat bounds for all deps :scream:

1 Like

Is there still a 3-day waiting period for new packages?

For version updates, is the new version merged once the checks complete, or is it a process that is run periodically?

I think it’s “yes” for both questions and maybe this is useful: https://github.com/JuliaRegistries/General/blob/35a193ca487155b7c4b9ececba086f579e13df61/.travis.yml#L18

2 Likes

You can retrigger if you want but I will go through the backlog manually later.

Yes.

The regular CI run approves the PR and then a cronjob merges. I set it to run every hour for now since the influx of PRs that meet the requirements is at the moment very low.

3 Likes

Is it possible to add very similar sounding malicious packages that way?

2 Likes

We’ll be implementing this: https://github.com/JuliaLang/Pkg.jl/issues/931. That’s also part of why there’s a three day waiting period. But adding a check that the edit distance to other package names is not too small is a good idea. Issue opened:

https://github.com/JuliaRegistries/GeneralRegistryCI.jl/issues/10

6 Likes

What about stdlibs? They typically do not have version entry (except Pkg.jl?):

Should their versions be identical to [compat] for julia?

1 Like

stdlib dependencies do not need compat entries.

4 Likes

Thanks for the clarification.

But wouldn’t it be a problem if:

  • packages upper-bound julia version as < 2,
  • stdlibs are decoupled from julia distribution, and
  • the packages start lower-bounding stdlib?

In this scenario, IIUC, older packages without stdlib bounds can be chosen by the dependency resolver. Not that this is the worst problem, since there are so many packages without upper bounds. But my impression was that start requiring upper bounds was for avoiding this kind of situation and making Julia ecosystem more robust.

You can retroactively add compat bounds to old versions of your packages. Just make a pull request directly to the General registry.

Perhaps we should open an issue to discuss whether we should require compat entries for stdlibs. Can you open an issue here: https://github.com/JuliaRegistries/RegistryCI.jl/issues

Sure:

https://github.com/JuliaRegistries/RegistryCI.jl/issues/19

2 Likes

BTW, it took me for a while to realize that this is the case. I think it might be very surprising for people coming from other language ecosystems. It would be nice to document it somewhere. Maybe Pkg.jl documentation? But it’s not strictly about Pkg.jl and presumably, each registry can have a different policy for this.

I’m also a bit worried about this flexibility. It would mean that Manifest.toml generated at some point in time can become “invalid” in the sense it becomes impossible for Pkg.jl to generate the same Manifest.toml (even when there is no new versions are released). Can we get around this by enforcing some kind of monotonicity in the compat bounds change in JuliaRegistries/General (e.g., only increasing bounds is allowed)? Another worry is that this probably is not going scale once JuliaRegistries/General becomes (say) as big as PyPI.

1 Like

Why is that a problem?

@StefanKarpinski I noted a few potential issues in last comments, but assuming that you are referring to:

I think this would also mean that, a Project.toml with some “tight” [compat] that is Pkg.instantiateable at some points in time may not be so at other points in time. I find it contradictory to the emphasis on immutable aspects in the rest of Pkg.jl. Also, I suppose it means that Pkg.resolve is not an idempotent operation? I think is a reasonable assumption one can make, even though it’s not documented to be so.

Maybe it’s OK since full reproducibility still requires Manifest.toml anyway. But, if JuliaRegistries/General “officially” allows tweaking version bounds, I think it’s nice to document these properties.