Julia's Release Process

Internals & Design
1

I wrote a blog post explaining Julia’s release process, collecting information from various places:

Also being discussed on Hacker News right now if anyone cares to join that conversation.

31 Likes
2

Thanks for making a blog post about this.

I am wondering if at some point there could be some community consensus or advice about packages depending on a minimum specific (minor) version of Julia. Applicable scenarios include

  1. relying on something specific that that was introduced in 1.x and just makes life easier and may not be available even in Compat.jl,

  2. performance/inference problems with earlier versions, but with code running correctly for all of 1.*

I think this is relevant because if at some point commonly used packages start depending 1.1 or 1.2 etc, users using 1.0.* will just have to make a choice between upgrading Julia or not getting improvements/bugfixes for some packages.

3 Likes
3

Very well written and very informative! Thank you.

And this scenario is what I during the last years constantly was being confronted in my R environment. It is even more drastically because I am repeatedly forced to upgrade the R version for a specific package which must be used in a specific minimum version and another must-have-package forbids the upgrade = deadlock, with various bad and errorprone not-solutions (workarounds).

Currently there is no Julia strategy to avoid similar deadlocks in future and I have no good solution. Julia and its package environment is community driven and thats what comes with it.

The big advantage, which let me don’t loose faith, is, that a Julia package is written in Julia and not C. Typically.

1 Like
4

Julia tends to be a lot easier to write code that supports a range of incompatible versions in. Compat is of course the ultimate version of that, but it can be done in a more one-off fashion as well. I think we’ll have to collectively figure this out as it becomes an issue. It’s a bit too early in the lifetime of the language to discourage people from using new features in new versions. I hope that all our diligence with making upgrades truly non-breaking will pay off by encouraging people to upgrade regularly without fear.

4 Likes
5

Hopefully it was not me who sounds discouraging, but I am afraid that I was, so let me say, just in case, the opposite is the case, it is not only that I do not loose faith, the truth is, that I am quite sure, that even if I run into version deadlocks in future, I will be able to solve these issues in a much better way than it is feasible in R. Solutions which are really solutions and not only workarounds.
And even now I am inexact, because R itself is a minor problem, it is bioconductor which is the main source of headaches. But take this just as a side note.

2 Likes
6

Is there going to be a way of bringing the 1.3 multi-threading stuff to Compat? To me, that seems a likely point for many packages to start dropping support for old versions.

1 Like
7

Definitely not. That’s not a thing you can bolt on.

1 Like
8

Somewhat tangential, but it’d be nice if there is a detailed write up like this for what it means to be a “public API.” I hope I don’t sound nitpicky, but without clear public API, a software is not really SemVer compliant. In particular, I think there are things that should be clearly documented that they are not stable API (output (but not the calling convention) of show, type parameters of public strcts, etc.). I know Julia has minor change as a solution for something like this (and clearly explained in the blog post) but I think discussion in GitHub PRs and issues would be greatly simplified if it does not involve running PkgEval.

9 Likes
9

This is what I am counting on implicitly, so I just depend on 1.1 and then 1.2 when I need it.

2 Likes
10

Maybe @non-Jedi meant that Compat could include no-op functions/macros which would allow writing code that is threaded on Julila 1.3, but single-threaded on earlier versions? I guess that would be doable?

3 Likes
11

That could work. Good idea! It also matches the “sequential projection property” that we’re considering making a requirement for threaded code, which is basically that code should not deadlock if you delete all parallel macros and just run sequentially.

2 Likes
12

My understanding is that “public API” is defined by export. Is it too narrow?

2 Likes
13

Yes. Consider e.g. JSON.parse or Meta.parse, which definitely are part of the public API, yet aren’t exported.

14

or Base.@propagate_inbounds or Base.@pure…

I also want to add that it is not only too narrow but also not enough. For example, show(io, "text/plain", x) pretty-prints x in a human readable form. But is Base/stdlib allowed to improve readability? If the exact output is documented to be a stable API, it is impossible. (See also: Version doctests · Issue #1050 · JuliaDocs/Documenter.jl · GitHub)

Another example is that, say there is a custom vector type in a stdlib:

struct CustomVector{T} <: AbstractVector{T}
    ...
end

is it backward compatible to add one more type parameter?

struct CustomVector{T, S} <: AbstractVector{T}
    ...
end

It is compatible for user defining function f(::CustomVector{Int}) but it is not for the usecase like

struct MyStruct{T}
    x::CustomVector{T}
end

because the field x will be boxed when a type parameter is added. IMHO Julia documentation should explicitly say that parametrized struct type must never be considered concrete. This means that you need to write

struct MyStruct{T, X <: CustomVector{T}}
    x::X
end

when performance matters. It also would be nice that each struct docstring defines what type parameters are considered public. The rule “parametrized struct type is not concrete” lets implementers hide some type parameters as implementation details. Another approach would be to say that only abstract types should be used for dispatch.

3 Likes
15

Yes, unexported symbols can be part of the API too, eg Base.IteratorSize, if they are documented.

Generally, packages also follow this convention: everything exported is part of the API, and in addition anything that is explicitly documented (in the docs, having a docstring is not enough). This is of course consistent with not exporting anything, just documenting a bunch of functions as your API. But there is no single formalized convention, so documenting what is and what isn’t part of your API is generally worthwhile.

4 Likes
16

Read it two days ago. Need a Chinese translation? I can help.

17

That would be great. The process would be to make a pull request to add the translated post to https://GitHub.com/JuliaLang/www.julialang.org/.

18

Done.

1 Like
19

Woah, who’s considering going to do what now? I’m also not sure that deadlock-avoidance would even fall out of that since that’s a concurrency bug, not a parallelism one. Whereas sequential projection means that parallelism bugs are UB (maybe detectable?), but I don’t know if they claim any limits on concurrency.

20

Talk to @vchuravy about Tapir.