Proposed release process and schedule

Internals & Design
,
1

Now that we’ve released Julia 1.0 and are close to 1.0.1 and have added some new features and minor changes to the 1.1 development branch (i.e. master), it seems like time to talk about the future of the Julia release process. After various discussions in person, on Slack, and on this week’s triage call, it seems like there’s fairly solid consensus behind the following plan.

Patch releases

  • Patch releases increment the last digit of Julia’s version number, e.g. going from 1.0.0 to 1.0.1 currently.

  • Patch releases, following SemVer, should only contain bug fixes, low-risk performance improvements, and documentation updates. Of course, what exactly constitutes a bug fix can be more subjective than one might naïvely imagine since people write code that relies on buggy behavior. In general, we try to be very conservative with patch releases and use PkgEval to ensure that there’s minimal risk. People should be confident that they can just upgrade to the latest patch release without worrying about it breaking things.

  • Patch releases should also avoid changing internals unless it is necessary to fix a bug. Even though changing non-public code is technically fair game in any release, we want to avoid it in the name of minimizing the risk associated with patch upgrades as much as possible.

  • Patch releases will be released approximately monthly (1.0.1 is a bit late by this schedule), unless there are insufficient bug fixes on the release branch to warrant a new release, in which case a month might get skipped.

  • About five days before a patch release is supposed to go out, we will run PkgEval on the backports branch; if it looks good, we’ll merge it and then freeze the release branch and announce that the release branch is ready for testing. If everything looks good after five days, the new patch version will be tagged.

Minor releases

  • Minor releases increment the middle digit of Julia’s version number, e.g. going from 1.0.2 to 1.1.0 (although it’s actually a bit more natural to ignore the last digit in this transition since one can release 1.0.3, for example, after releasing 1.1.0).

  • Minor releases may include bug fixes, new features and “minor changes”—which is the term we’re using for technically breaking changes that are sufficiently unlikely to break anyone’s code and which do, in fact, not break anything in the package ecosystem as determined by running PkgEval and verifying that there isn’t any breakage.

  • Minor releases are also where significant refactorings of internals go, since we should only be refactoring to the extent that is necessary for fixing bugs in patch releases. This means that if you’re relying on some internal Julia stuff that’s not public, your code might break in a minor release. This is allowed according to SemVer since the change isn’t to a public API—so technically it can break at any time; but we will avoid this in patch releases, so minor releases will be where you have to watch out if you rely on internals somehow.

  • Minor releases will be done every four months, which means that three minor releases per year. That means the next minor Julia release—1.1.0—should be out before the end of the year. I know that some people will say “yeah, yeah, we’ve heard that before”. But this is a totally different ball game. We’re not trying to get a lot of interconnected breaking changes into minor releases, which is what has caused release slippage in the past. We can now do timed releases: if a change is ready, it goes into the release, and if it isn’t, it will just go into the next one.

  • It’s an open question how many release candidates we’ll need for minor releases. We’ve needed about 4 release candidates for releases in the past, but those were major releases with lots of large breaking changes to test. We may be fine with just 2 or 3 release candidates for minor releases. To be conservative, we should consider cutting 1.1.0-rc1 around November 15th so that we have time for a full month of release candidates before 1.1.0 final.

Major releases

Who knows? This is still a big open question. Since we only just got 1.0 out, it’s a bit early to start thinking about what 2.0 means. Ideally, we’d like to allow people to opt into breaking changes in minor releases with something like Python’s from future import feature. But the details of such a capability are yet to be determined. Other breaking changes, like modifications to how subtyping works (which has already been discussed to get rid of some less-than-ideal corner cases), can really only be all-or-nothing and cannot be opted into in a local fashion. We’ll won’t worry about crossing that bridge until we’re much closer to it.

Long term support

There was a lot of discussion on this week’s triage call about how to handle long term support and backporting bug fixes. If we had infinite resources, we’d backport every bug fix to every old release branch it applies to. Realistically, however, we don’t really have the capacity to maintain more than two active backport branches at a time. Backporting to the oldest version around seems wrong since then people who want stability are actively discouraged from upgrading or using new features ever. So the compromise that we converged on in the triage discussion was to have three active branches going at any time:

  • master where all development work happens, from bug fixes to new features and minor changes (and eventually, breaking changes).

  • The stable release branch: the release branch with the highest major/minor version number. Currently this is release-1.0. The stable release branch gets all applicable bug fixes backported to it from master.

  • A “long term support” branch, which is older than the stable release branch. Currently there is no such branch, but in the future, this will also get applicable bug fixes backported to it form master; it may also get its own versions of bug fixes as necessary.

The master and stable release branches are pretty clear. The real question is which past release branch should be the long term support branch at any given time and when does it change? The best answer we have at the moment is that we’ll play it by ear and see what people seem to be using. Historically, we know that 0.3 was very stable and a lot of people used it for a long time and it had 12 point releases, so the 0.3 release seems like a role model release for the “long term support” concept. Once 1.1 comes out, 1.0 will be the new LTS branch since it’s the only other release branch; but after 1.2 comes out, we have a choice between keeping 1.0 as the LTS branch for longer or making 1.1 the new LTS branch. We’ll have to see how people feel and what the demand is.

If some person or organization has a vested interest in keeping a particular older release branch going and is willing to contribute the work to make that happen (cherry-picking backports and kicking off PkgEval runs to make sure things aren’t broken), we’re more than happy to accept that help and make more releases, effectively make it an additional LTS branch. So you can always get longer term support by doing the maintenance yourself (or paying for someone to do it); but others should keep in mind that this won’t be official and if whoever is doing the maintenance stops doing so, there is no official guarantee of continued maintenance. As we work on automating all of the backporting and release processes going forwards, we may find that it’s sufficiently easy to maintain older branches that we increase the number of long term support branches that we can maintain.

Risk Tolerance Personas

Most users will fall into one of the following four categories of risk tolerance:

  1. High risk tolerance. YOLO, I live on master. Of course, this isn’t as risky as it used to be since there won’t be breaking changes on master for a while, so packages should continue to work even on master, but bugs happen, y’know? I’m willing to help find them.

  2. Normal risk tolerance. I like things to work and don’t want to deal with transient bugs on master. So I’ll stick to the latest stable release and upgrade to the current patch release of that when it’s available since that’s pretty safe and I’ll get bug fixes and perf improvements. The only annoyance is when a package I’m using breaks because it was depending on some Julia internals and it may take a while before it gets a new release.

  3. Low risk tolerance. I’m conservative and risk-averse. I follow the current long-term-support branch which has gotten significant testing and is already at least on its third or fourth patch by the time it becomes the long-term-support branch and I start using it. By the time I switch to a release, any package breakage there might initially have been has long since been sorted out.

  4. Very low risk tolerance. I’m extremely risk averse. I never upgrade Julia (or anything) except for critical bug fixes and security issues. I run a version of Julia that’s no longer actively supported, but it’s the last release of a former long-term-support branch so has a double-digit patch number and has been really thoroughly debugged. If I need a new bug fix on this ancient release branch, I will backport it myself and help cut a new, even more reliable patch release.

These profiles make it a bit clearer that the main criteria for the long-term-support branch are that the branch has these properties:

  • It has had sufficient patch releases that we’re confident that it’s highly reliable;
  • Packages that are ever going to support it have already released versions that do.

If these two criteria are satisfied by a new long-term-support branch, then users in the “low risk tolerance” category will be able to upgrade to the new LTS branch since they can already be confident that it will be reliable and well-debugged and that packages they need will be ready to use (although they may need to upgrade their versions of packages). We’ll have to learn from experience how many releases the long-term-support branch should lag the stable release branch by.

Questions, comments and suggestions are welcomed.

47 Likes
2

Just to clarify, will our versioning going forward be in accordance with SemVer?

If so, we shouldn’t say “technically breaking” here, right? That language implies (to me, anyway) that we’d be releasing backwards-incompatible changes to the public API, which is disallowed for minor releases under SemVer AFAIK.

After reading the following bulletpoint (Minor releases are also where significant refactorings of internals go...), I went back and realized “technically breaking” above could’ve meant “changes that could break people’s code if they are relying on internals, but will not break their code if they are following the public API.” Is that the case? If so, that seems in accordance with SemVer.

Note that I’m cool if we don’t follow SemVer. If we deviate from it, though, I think we should have explicit explanations for each deviation. If we, in fact, do follow SemVer, I think that should be stated explicitly as well (I see it was mentioned explicitly for patch releases, at least).

3

We will be following SemVer. However, what is or isn’t considered a breaking change is inherently a bit of a judgement call. Technically, any bug fix is a breaking change since people can and do write code that depends on buggy behavior. In order to guarantee not breaking any possible code that someone could have written, one cannot change anything at all—which clearly isn’t reasonable. (And if that’s what you want, just don’t ever upgrade Julia or packages.)

In some languages, there’s public/private API enforcement, which helps, but Julia doesn’t have that, so we need to draw the line somewhere. The clearest way to draw the line is by testing whether a change actually breaks code or not. Fortunately, we have a very large corpus of code with tests: all the registered Julia packages. If a change seems very unlikely to break real code AND it does not break any packages, that’s pretty strong evidence that it’s a reasonable change even if it’s “technically breaking”. (There may be ways to extend this testing practice to private code as well, which we’ll be exploring in the future.)

The term “technically breaking”, while accurate, is a bit alarming. So we’re calling changes that are technically breaking but which don’t in practice actually break anyone’s code, “minor changes”. There are four minor changes currently slated to go out in 1.1:

  • #29061: consistently treat comma at end-of-input as incomplete.
  • #29092: convert values put into channels on put! instead of on take!.
  • #29211: make the printing of BigFloats consistent with other floating-point types.
  • #29274: make “stream unusable” error an IOError instead of an ArgumentError.

These are all technically breaking in the sense that one could write a program that relies on the old behavior and breaks because of the change. Is there likely to be any code in the wild that actually breaks because of these changes? No. In each case, any code that is broken after the change was almost certainly broken before the change as well. For example, if you have a program that puts the wrong type of value into a channel then that program was going to error anyway; now you get the error earlier than you would have before. Could changing the parsing of programs that end with a comma break something? Technically yes, but how many Julia programs out there end with a comma? Not very many, I’d wager. We’ll find out when we run PkgEval. We could always weasel word these changes and say that the old behavior was buggy and that we’re just fixing bugs. But calling these changers “minor changes” seems more direct.

4 Likes
4

Are there any thoughts how this process expands/interacts to/with external packages? How should I deal with packages like DataFrames if I’m a low risk tolerance user? (DataFrames pops in my mind because it was broken for a couple of hours this week). Is this all on the user to test every up in the package manager?

5

I think we should encourage packages to follow semantic versioning as well, which should help signal which updates are safe and which ones might break things. There’s some degree to which automated testing can help enforce that, but in the end it’s up to package maintainers to follow semver and on the end-user to test things.

6

Great, thanks for the explanation.

The term “technically breaking”, while accurate, is a bit alarming. So we’re calling changes that are technically breaking but which don’t in practice actually break anyone’s code, “minor changes”.

Following SemVer, any new release (even a patch release) can possibly break somebody’s code if it introduces a bug, or fixes a bug that downstream code accidentally relied on.

However, such breakages are not blatantly disallowed for minor releases under SemVer; SemVer uses the term “backwards incompatible” to describe changes which are disallowed for minor releases. Since we are following SemVer, I think it would be useful to adopt this terminology - rather than “minor changes” or “technically breaking” - in Julia’s official release/versioning specification.

In some languages, there’s public/private API enforcement, which helps, but Julia doesn’t have that, so we need to draw the line somewhere. The clearest way to draw the line is by testing whether a change actually breaks code or not.

A well-defined public API is required to follow SemVer. Documentation is an acceptable form for such a definition; should we specify Julia’s documentation as Julia’s official “public API” definition?

I think this is the case where adopting the parlance of SemVer can make these decisions easier/clearer. In SemVer-lingo, these are all “backwards compatible” changes: AFAICT, none of the changes alter behavior which was specified by Julia’s public API for the current major release (assuming public API == Julia’s documentation). Thus, we can say that we are allowed to put these changes in a minor release.

Of course, committing to extra reverse-dependency testing (e.g. PkgEval) before doing a release is awesome as well. It just doesn’t play into versioning decisions under SemVer. If we want PkgEval to play a roll in our versioning decisions, we could add an addendum like “changes which break more than X% of the ecosystem, even if they are technically backwards compatible, are not allowed into minor releases” or something in that vein.

3 Likes
7

I am also confused by the precise meaning of “public API”. Is this the set of exported symbols? Or as @jrevels suggests, the set of documented symbols? Or the union of both?

1 Like
8

I don’t think that’s desirable, as it could discourage people from writing documentation.

1 Like
9

I think this topic is only about Julia’s code itself.
Other packages/projects can adopt different definitions.

10

But even Julia language developers are people and can get lazy sometimes :wink:.

1 Like
11

Which of these various releases and pre-releases will have binaries available?

12

Also, I believe SemVar says anything goes with 0.x releases. So if you are risk adverse don’t use any 0.x packages (such as DataFrames).

1 Like
13

What exactly constitutes the public API of Base code needs to be spelled out in greater detail, but the convention at this point is mostly around export: if something is exported, then it’s public, if it isn’t, even if it’s documented, it should be presumed to be internal—although the documentation really should say that. You’re definitely safe if you stick to things that are both exported and documented.

1 Like
14

I’m not entirely clear what you mean by this question. All releases have binaries. Release candidates, alphas and betas (where relevant) also have binaries. The only things that doesn’t have a corresponding binary release is the warning announcement before a new patch release.

15

This sounds good to me. Does this versioning include stdlib?

At some point I seem to remember it being suggested that we could use different versions of stdlib with different versions of Julia. (E.g. get the latest base but keep an old version of some stdlib package if necessary.)

16

That will be possible at some point but we don’t have the technical ability to do that yet. We won’t make breaking changes to stdlibs until we can though.

17

Perhaps I missed the point of Python’s future (totally possible, I’m not very into Python world), but I’ve always seen Compat.jl as its Julia’s equivalent, isn’t it?

18

For what it is worth, I thing a lot more people would test patch pre-releases if you provided binaries. Ideally, these binaries could just become the official patch binary after a suitable amount of time.

19

Release candidates do and always have had binaries. So have alphas and betas. And there are also nightly builds in general.

1 Like
20

He is referring to patch releases. The recent “Test 1.0.1” post did not mention any binaries but only building the corresponding branch yourself. But is this not some kind of release candidate?