New automerge requirement: Release notes required for breaking package releases

[ericphanson]

When registering a breaking release of a package, you now need to add release notes to the registration that mention what the breaking changes are, so that your users have an easier time understanding what they need to change in their code in order to use the new version of your package.

The release notes must include the keyword “breaking” (or “changelog”), because the intention is for you to describe what the breaking changes are, or where to find them, to help your users update their code.

What is a breaking release?

Julia’s package manager Pkg.jl uses semantic versioning, often shortened to semver, to manage the compatibility between packages (see 6. Compatibility · Pkg.jl for the specific notation and rules), meaning version numbers take the form $major.$minor.$patch. If you declare compatibility with 0.2.3 of a package via MyDep = "0.2.3", Pkg will allow upgrades for any 0.2.x patch (for x\geq 3), and likewise for MyDep = "1.2.3" any upgrade up to (but not including) version 2. A breaking release is in (Julia) semver terms is a minor version change when pre-v1, and a major version change post-v1.

How do I add release notes to my package registration when registering with Registrator.jl?

See Registrator.jl’s documentation here. In short, you add them just after the @JuliaRegistrator register command. If you forget them, you can retrigger registration again and add them and it will update your PR automatically.

How do I add release notes to my package registration when registering with JuliaHub?

Enter them in the box labeled “Release notes” when you register:

What if I use a changelog to describe my breaking changes?

You still need to enter some form of release notes, but you can simply point to your changelog. E.g. “See CHANGELOG.md for a description of changes in this release”. (Your release notes need to mention “breaking” or “changelog”, case insensitive, to pass the check).

What if I make a huge number of breaking changes and can’t mention all the changes?

You could summarize the most important points, e.g. “The main breaking change is a renaming of the type MyType to MyType2. There are other breaking changes but you need to look at the code changes to find them all”. This is discouraged since it is less helpful to your users, but would suffice to pass the automerge check.

Where do the release notes show up?

Registrator (via comment bot or juliahub) will add the release notes to the registration PR. Then if you have TagBot configured for your repository, once the version is merged, it will add those release notes to the git tag / github release it creates.

Where are all the AutoMerge requirements listed?

They are here: Automatic merging guidelines · RegistryCI.jl

See also General’s README which links to them and also contains a useful FAQ for package registraton.

Do I need to comply with all AutoMerge requirements for my package to be registered?

No. Complying with AutoMerge’s requirements is highly recommended because it means your registration can be processed automatically without manual intervention, and because it helps meet some basic best-practices. However, if you cannot meet the requirements for some reason, you can ask for your registration to be manually merged. Make a request in the #pkg-registration channel on the public Julia Language Slack, or directly on the registration PR itself (or both).

I don’t like this, can we not?

We can change this rule if it proves to be unduly burdensome. Let’s try it out.

This check is great, can we add more?

This check came about since @ianshmean kindly filed a PR to implement it, and it seemed reasonable to the registry maintainers who reviewed it (e.g. Dilum reviewed and did not leave a blocking comment, and I approved it). If you would like to add a check, give it a try!

Note that we try to balance burden to developers and benefit to the ecosystem, so if your proposed check is much stricter than the current rules, it may be worth an issue or discussion here on Discourse to discuss it first.

[palday]

It might be good to support “NEWS” in addition to “changelog” since “NEWS.md” is also a common place to put this stuff.

[ericphanson]

Yeah, that could be handled currently by writing “See the changelog in NEWS.md”, but maybe it’s worth a special case.

[mofeing]

While I agree that there should be some changelog when there is a breaking release, I’m not sure I like it for pre-v1 releases since by nature it means that code is not yet stable and breaks can be common.

[goerz]

This requirement should not be active for pre-1.0 releases. SemVer is very specific about the meaning of pre-1.0 versions. By definition, pre-1.0 versions do not have a stable API. Asking people to describe “breaking” changes waters down this distinction between pre-1.0 and post-1.0 version. I would actually consider it a violation of SemVer.

This actually also includes the v1.0 version itself: this is the first version that defines the stable API, so it arguably can’t have “breaking” changes. I’m fine with requiring release notes for v1.0, I’m just concerned about

@ericphanson

Your release notes need to mention “breaking”

A good release note for v1.0 is “First stable release”. Just like good release notes for v0.1 are “First public release”.

Pre-1.0, I’m not sure it’s a good idea to require release notes at all, and certainly we should not require labeling any change as “breaking”. I’m fine if people want to have a quasi-stable API pre-1.0 with a changelog, and I’m also aware that Pkg treats v0.x versions as “incompatible” (which I think is pragmatic in light of SemVer not imposing a specific meaning on v0.x vs v0.x.y releases). But we shouldn’t enforce post-1.0 semantics on pre-1.0 versions. As the SemVer spec says, “If you’re worrying a lot about backward compatibility, you should probably already be 1.0.0”. Asking for changelogs pre-1.0 undermines that, and only serves to proliferate ZeroVer packages, which I think are a bad look for the ecosystem.

Let’s encourage packages making v1.0 releases when they are, in fact, stable, but let’s also give pre-1.0 releases their proper role as the “experimental” part of the development process that should not be burdened by improper red tape.

[goerz]

Actually, if we want to encourage the proper use of SemVer, it would make sense to require release notes on all releases >=1.0, not just breaking releases. That’s exactly the difference between pre-1.0 and post-1.0: SemVer requires that post-1.0, you properly distinguish breaking changes, feature changes, and bug fixes. Pre-1.0, such distinctions are optional.

[j-fu]

However, the Julia package manager enforces post-1.0 semantics on pre-1.0 versions, so for me, the current requirement appears to be quite logical.

As for requiring release notes for every version after 1.0 - I think this might be a good idea for minor releases, not sure for patch releases. Also, this might be implemented at a later stage, when we see how the this feature works. I personally started to introduce changelogs into my packages, I think this is the most convenient way.

BTW, git-cliff is a nice tool to start one from git commit messages.

[ericphanson]

I think with the current Julia ecosystem, there are a bunch of heavily used, infrequently breaking packages that are nonetheless pre-v1. In fact, here’s 100 packages that are pre-v1.0 but have added an average of > 140 new users per day for the duration we have package statistics collected:

JSON, DataStructures, LogExpFunctions, MacroTools, DocStringExtensions, StatsBase, TranscodingStreams, IrrationalConstants, CodecZlib, InverseFunctions, ColorTypes, ChangesOfVariables, Distributions, GR, Colors, ForwardDiff, Latexify, Formatting, FixedPointNumbers, PDMats, RecipesPipeline, ColorVectorSpace, Contour, IniFile, StructArrays, FFMPEG, Measures, CommonSubexpressions, Rmath, Distances, UnicodeFun, GPUArraysCore, ExprTools, TensorCore, Parameters, CSV, FilePathsBase, HypergeometricFunctions, Unzip, Calculus, BitFlags, Interpolations, DualNumbers, GeometryBasics, Inflate, Ratios, CEnum, IfElse, SimpleTraits, AbstractTrees, LoopVectorization, TimerOutputs, ArrayInterfaceCore, StringManipulation, Observables, ArnoldiMethod, VectorizationBase, JLFzf, IntervalSets, DensityInterface, SLEEFPirates, CPUSummary, ZygoteRules, CommonSolve, LayoutPointers, JLD2, NearestNeighbors, HostCPUFeatures, PolyesterWeave, ThreadingUtilities, CloseOpenIntervals, KernelDensity, BitTwiddlingConvenienceFunctions, JuliaInterpreter, PositiveFactorizations, MappedArrays, SymbolicIndexingInterface, Widgets, ManualMemory, Clustering, CodecBzip2, Mocking, ExceptionUnwrapping, SIMDTypes, Tricks, ImageCore, Extents, CategoricalArrays, PaddedViews, CompositionsBase, CpuId, StrideArraysCore, Polyester, TiffImages, Transducers, PreallocationTools, PNGFiles, NNlib, RecursiveFactorization, Lazy

I think a breaking change to almost any of these would be fairly disruptive and users would benefit from release notes that indicate what the breaking changes were. There are 7k more pre-v1.0 packages as well.

code

using CSV, DataFrames, RegistryInstances, UUIDs, Dates

registry = only(filter(reachable_registries()) do reg
    reg.name == "General"
end)

# Download and uncompress:
# https://julialang-logs.s3.amazonaws.com/public_outputs/current/package_requests.csv.gz
requests = CSV.read(expanduser("~/Downloads/package_requests.csv"), DataFrame)
subset!(requests, :client_type => ByRow(==("user")), :status => ByRow(==(200)); skipmissing=true)

function lookup_pkg(uuid)
    pkg = get(registry.pkgs, UUID(uuid), nothing)
    pkg === nothing && return (; name=missing, latest_version=missing)
    name = pkg.name
    latest_version = maximum(keys(registry_info(pkg).version_info))
    return (; name, latest_version)
end

versions = unique(select(requests, :package_uuid))
transform!(versions, :package_uuid => ByRow(lookup_pkg) => AsTable)

leftjoin!(requests, versions; on=:package_uuid)
disallowmissing!(requests, [:name, :latest_version])
subset!(requests, :name => ByRow(!endswith("jll")))

counts = select(requests, :name, :latest_version, :request_addrs, :date_min, :date_max)
transform!(counts, [:request_addrs, :date_min, :date_max] => ByRow() do n, min, max
        n / Dates.value(Day(max - min))
end => :request_addrs_per_day)
subset!(counts, [:date_min, :date_max] => ByRow(!=))
sort!(counts, :request_addrs_per_day; rev=true)
pre_v1 = subset(counts, :latest_version => ByRow(<(v"1")))

println(join(first(pre_v1.name, 100), ", "))

I agree more packages should move to v1+, but I don’t think adding additional restrictions for v1+ helps that. If anything, it makes an incentive not to move to v1, since then there’s additional rules to follow.

That being said, I’m not totally opposed to relaxing this rule. But my biggest concern with it is burden to package developers, not principles of semver.

What are the main issues with requiring release notes? It is a new workflow to some, that’s true, but JuliaRegistrator has been prompting for release notes for years, and JuliaHub has similarly had a textbox for it for a long time. Is it coming up with something to say? Or a mismatch between how folks are already doing release notes in some other way?

[ericphanson]

One other point I want to make here: the purpose of General is to share code you have written with users, pre-v1 or not. If a package is not meant to be used yet, it does not need to be registered. So I don’t think it is so bad to have some small friction in breaking releases, asking for some release notes to say what has broken.

I don’t think it should be “overly” burdensome, and it should not meaningfully reduce the development velocity of julia packages, and this particular rule may need to be adjusted or removed if it causes such problems. But on its face, asking a developer to spend a minute or two listing key breaking changes in a breaking package release (which may help save many more users time to dig into what those changes are) doesn’t seem like it.

[mofeing]

@ericphanson

What are the main issues with requiring release notes?

The main burden for me is that users IMO will check the docs or the repository’s README for checking breaking changes, not the JuliaHub frontend.
And as a developer, I don’t want to be copy-pasting everywhere.

Actually, I don’t post somewhere the breaking changes but reflect it properly in the docs. Like using @deprecated, putting info boxes, … Specially in post-v1.

This is a very personal opinion, but I think this kind of info should be written only in one place, and that place is the docs.

@ericphanson

In fact, here’s 100 packages that are pre-v1.0 but have added an average of > 140 new users per day…

Yeah, but the solution to this is to propose them to move to v1

@ericphanson

If anything, it makes an incentive not to move to v1, since then there’s additional rules to follow.

I think that this will make people register less packages on the General registry. With the new source feature, the General registry is less needed.

IMO the solution passes by having a more diverse set of registries, not pushing everything into General, and leaving General with relaxed rules.

ps: Now that we have public, checking breaking changes is probably automatizable, right?

[ericphanson]

The main burden for me is that users IMO will check the docs or the repository’s README for checking breaking changes, not the JuliaHub frontend.

The release notes created by tagbot show up in the git tags in the repository, and in github releases, so the package repository. I don’t know if JuliaHub surfaces it or not.

And as a developer, I don’t want to be copy-pasting everywhere.

You don’t need to! Just write “the breaking changes are documented in the README”.

[Datseris]

@j-fu

However, the Julia package manager enforces post-1.0 semantics on pre-1.0 versions, so for me, the current requirement appears to be quite logical.

Exactly.

@ericphanson

I agree more packages should move to v1+, but I don’t think adding additional restrictions for v1+ helps that. If anything, it makes an incentive not to move to v1, since then there’s additional rules to follow.

Exactly again.

I couldn’t agree more with the arguments given here by Eric, and as far as I can tell these arguments lead to a huge quality of life improvement for users.

I also find the notion of “pre 1.0 nothing matters” extreme, and unlikely to be helpful in the real world. I would say: as soon as package has users, you really should be taking changelogs seriously. Instead of focusing on what is the number of your version, focus on what actually matters: who cares about the changes you do to your software.

[Eben60]

@mofeing

he main burden for me is that users IMO will check the docs or the repository’s README for checking breaking changes, not the JuliaHub frontend.
And as a developer, I don’t want to be copy-pasting everywhere.

If they will, how many times they wouldn’t find there a single word about the changes, breaking or not?

And if your package has a changelog in it’s docs (how many do have it? 30% - or is it too optimistic an estimation?), is it really really a burden for you just to paste the link to the changelog into the box labeled “Release notes”?

@mofeing

ps: Now that we have public, checking breaking changes is probably automatizable, right?

It is probably possible to detect some breaking changes, but not any.

[Datseris]

@mofeing

ps: Now that we have public, checking breaking changes is probably automatizable, right?

I don’t see how this is possible. I can leave a function signature identical to what it was before, and make the function body do something totally different, so that the return type remains the same but its value changes dramatically. LIke, changing the cosine to calculate the sine instead. I don’t see how this can be automated, at least not simply.

[mofeing]

Yeah, you’re right. Breaking change on semantics is imposible to check, but at least breaking changes on the exposed interfaces are.

@Datseris

I also find the notion of “pre 1.0 nothing matters” extreme, and unlikely to be helpful in the real world.

I didn’t say that pre-v1 anything can done. What I said is that breaking changes should be expected.

@Datseris

I would say: as soon as package has users, you really should be taking changelogs seriously. Instead of focusing on what is the number of your version, focus on what actually matters: who cares about the changes you do to your software.

Sure, changelogs are nice, but we have versioned docs which also do the work.
Also, it can be a burden in the early stages of development because things change fast and sometimes, radically.

Also, I don’t think that user usage is the best metric. In the end, it’s the author who says if a package is ready to use or not. If people start using such packages, they should be aware of the risk. I know that that passes by the fact that a loooot of critical packages in the Julia ecosystem are <v1. This is not a problem new to Julia, it has happened in a lot of other languages. And the solution goes by trying to convince the authors of such packages to stabilize the code and do a v1 release.

I want to give an example I’m involved in right now. I’m now developing a package with some collaborators that it’s very experimental and we are breaking a lot of things all the time.
People are expectant of this package so they start using it even if it’s not ready for production. We are ok with it but we warn them that it’s still in early stages and should not rely a lot on it.
Due to different priorities (e.g. we want to submit to some journals, conferences, …), the docs are missing a lot of things and even tracking the breaking changes is hard (like really, we are working full time and breaking lots of things between versions).
We registered it in General because we need to test integration with some other packages or have a very small selected set of users who we want them to try it.
We do breaking minor releases, but mostly for compatibility with the other packages we are integrating with.
Of course, when it’s ready for production, we will impose better practices, document everything, … but right now is not gonna add much value.

[adienes]

@mofeing

Yeah, but the solution to this is to propose them to move to v1

+1

If a package is pre-1.0, then that is a signal from the developers to users that the API may not be stable. Period.

Don’t let Hyrum’s Law win! as a user, don’t use a package <1.0 if you’re not willing to occasionally face breaking changes. as a developer, don’t feel pressured by a userbase to not improve your package with breaking changes if you haven’t released a stable version yet

Otherwise the distinction is just fully meaningless, and it just becomes an arms race to add more and more decimal places…

[giordano]

@adienes

as a user, don’t use a package <1.0 if you’re not willing to occasionally face breaking changes.

The problem isn’t facing breaking changes. It’s not having a clue of what changed and how to adapt to the changes. That’s orthogonal to being “stable” (for some definition of “stability”).

[goerz]

@Datseris

I also find the notion of “pre 1.0 nothing matters” extreme

I don’t know how often I have to say this. It’s not extreme, it’s literally what SemVer says. It’s an important part of the specification. If we’re just throwing that overboard, we should not claim to follow SemVer, because we’re just not.

I find it both puzzling and alarming that there are core members of the community (including registry maintainers) who just choose to willfully ignore the role that unstable versions play in SemVer.

I didn’t really have a problem with Pkg treating v0.x versions as incompatible. But seeing that this has apparently caused people to dig in on the mistaken notion that 0.x and x.0 are semantically equivalent has actually made me think that it was a huge mistake. It would have been much better to treat 0.x the same as 1.x (i.e., as “compatible”). Alas, that ship has sailed.

It appears I’m in the minority with holding up SemVer, so I guess I’m going to have to let it go. I still think it’s causing significant harm to muddle up the distinction between stable and unstable releases. People should start with v0.1 and then switch to v1.0 much sooner than they typically do. And, we should enforce quality standards >1.0 to that extent that that is technically feasible: A v1.0 release needs full documentation of what constitutes the stable API, tests and a changelog in some form.

@Datseris

as soon as package has users, you really should be taking changelogs seriously.

Arguably yes, but then you should also switch to v1.0. It’s a bit of a chicken-and-egg problem: You want to (or at least I want to) make code public while it’s still “experimental”, so that you can make sure that the API survives contact with real-world usage. Certainly, if a package has many users/dependents, it should be stable, and keep a full changelog. When exactly to switch from v0.x to v1.0, or whether you want to start with v1.0 at registration (with all that entails), is up to every individual developer to decide. I also do recommend having some release notes for any release, stable or not. But that doesn’t mean we should have tooling that forces a changelog with incorrect breaking/feature/bugfix semantics. At that point, I’d just opt out of General for unstable packages, personally.

[goerz]

@giordano

The problem isn’t facing breaking changes. It’s not having a clue of what changed and how to adapt to the changes.

That’s exactly true, and why I’m not at all opposed to forcing release notes for any version. I’m also fine with the new requirement which enforces this for releases that Pkg considers incompatible (both 0.x and x.0).

I would be opposed to having further requirements beyond “release notes exist”, at least pre-1.0. In particular, don’t require that words like “breaking” or other post-1.0 semantics appear.

Beyond that, the only other consideration is whether the new requirements are practical. Personally, I’ve never using release notes during registration (when commenting @JuliaRegistrator register to initiate a new version release, I mean). Instead, I use the release notes on GitHub (partly auto-generated) for pre-1.0 packages, and a full CHANGELOG for post-1.0 packages in addition to that. I’m open to changing my workflows, though, and I’m definitely in favor of the idea that we should enforce that when packages make releases, they should document in some form what changed and how to adapt to that change. This is independent of whether the release is stable or not.

[Datseris]

@goerz

I don’t know how often I have to say this. It’s not extreme, it’s literally what SemVer says.

SemVer doesn’t say “pre 1.0 nothing matters” at all according to my understanding of it, but I think we are being pedantic now and it is not productive to argue about what “exactly” should be done with pre-1.0 versions according to SemVer.

In my view what is important and productive I think is to focus on the common grounds:

1. Stating somewhere (preferrablu in a dedicated, human-readable changelog) what changes are done to the software, even for minor changes, and
2. Motivating people to release 1.0 faster/sooner if their package has users.

I believe the proposal here hugely helps with # 1.