1.0 progress/status

I figured I’d write a post updating people with where we’re at on getting Julia 1.0-alpha ready. We’re several months overdue at this point, so people may be getting impatient. I know that I am and I feel confident that I speak for all Julia core devs – we can’t wait to get this release out the door.

The major work item that’s holding things up at this point is a significant rewrite of the optimizer that @keno is working on. It’s already fairly far along, but still not done. Why is he rewriting the optimizer, you ask? Excellent question. The proximate answer is that we want to change the way the iteration protocol works (motivation,
julep, pull request) and the new iteration protocol is dog slow without a new and much improved optimizer. Changing the iteration protocol to use a nothing | (value, state) style instead of the current (value, state) style has revealed that @jameson’s excellent work on supporting these kinds of unions efficiently in the first place got us to this point, but when you push it too hard – which the new iteration protocol does – the optimizer can’t handle it. Hence @keno’s new optimizer, which we needed at some point anyway but were planning on doing in a later 1.x release.

We could have lived without the iteration protocol change in 1.0, however, so the question remains: why are we doing this now, right before 1.0? The real reason that the new optimizer is crucial right now is that we’ve decided on using unions for missing data in 1.0. Now that we’ve started using nothing | value and missing | value all over the place in Base and in the data ecosystem, we’ve found that we’re on the precipice of the old optimizer falling apart anyway. The iteration protocol change reveals this precarious position, for example, when iterating a Union{Missing, Int} vector: the new iterate function has a return type something like Union{Nothing, Tuple{Union{Missing, Int}, Int}}. This just ends up being too much for the poor old optimizer. However the key point is that iteration is just the canary in the union coal mine – now that we’ve embraced small unions, these kinds of nested unions are inevitably going to crop up. Fortunately, the new optimizer handles these kinds of constructs without complaint – and almost incidentally makes the new iteration protocol fast. So the new optimizer is necessary both to change the iteration protocol (nice but not crucial) and to make sure that the new data ecosystem works as well as promised (crucial).

While that’s going on, @mbauman has been making a series of PRs improving generic array code and broadcasting. Every PR that Matt makes is, in my mind, another a significant chunk of array brokenness that we would eventually have regretted that we now won’t have to live with until Julia 2.0. So I’m glad the optimizer work is giving him some time to get it done before 1.0-alpha.

The rest of us core devs are looking around for various things that might be suboptimal, debating if and how to improve them, and fixing them. See the triage label for whatever’s on the table at the moment. Currently there’s only one item in there, which frankly, feels great :smiley:.

TL;DR: we are very close to being ready for 1.0-alpha and the work that’s delaying it is appropriately important, although as is often the case for programming language development, somewhat obscure.

128 Likes

I think I speak for everyone here when I say that I applaud the Julia devs commitment to “getting things right”, even in the face of deadlines and legacy influences of other languages. The payoff of this attitude is already abundantly evident in this beautiful language.

As someone who spends a lot of time in the data ecosystem I can tell you that the improvements to Union types are paying off big time: having performant Union types drastically simplifies quite a lot. I’m sure we all remember the bad-old-days of NullableArrays.

One question I have is how much thought has been giving to more extensive compile caching and other compile-time improvements. Compile times seem significantly slower in 0.7. As a former C++ person I’m pretty tolerant of this, but I have to admit that this is starting to chafe even me in some cases (certainly plotting will be in very rough shape until the situation improves, unless someone comes along and writes a drastically different plotting package). Certainly I can make things better for myself by changing to a work-flow that requires less re-initialization, but I suspect this issue will loom larger and larger as time passes. (Of course I know this is not going to be addressed in 1.0.)

27 Likes

Well, I am actively working on Makie & PackageCompiler which should be a good answer to you concerns. Makie release will likely happen around the Julia 1.0 release!
While we still won’t have perfect, incremental, binary caching, you will be able to at least cache most binary for the biggest offenders!

25 Likes

Compile time improvements are going to be a very important focus very soon. You answered your own question though — we aren’t doing it right now (pre 1.0) since speeding up the compiler doesn’t change the language.

I hope we’ll be able to put a lot of effort into this after 0.7 is released. Some groundwork has already been laid, such as improving the interpreter and some of Jameson’s restructuring of the code generator.

19 Likes

While I am looking forward for Julia 1.0 like I am sure everyone else here, I believe it is far better to wait some more time instead of leaving open issues such as inefficient Union types etc, which would influence packages development and therefore would be much more difficult to get rid of after 1.0 release.
Maybe while the new optimizer is developed there could be some more time to reconsider some minor language glitches such as this one or others which may have been postponed after 1.0 release due to time constrains.

1 Like

I also wanted to congratulate Julia core devs for the amazing work so far.

It would be amazing if the new optimizer could fix the performance issue of combining Missings with NamedTuples: see this thread for example, the issue was discussed many times.

2 Likes

I’m a bit confused by this response – the tone says “I disagree” but the content seems to indicate that you believe the approach we’re taking is the right one…

Maybe while the new optimizer is developed there could be some more time to reconsider some minor language glitches such as this one or others which were postponed after 1.0 release.

That was deemed to be a documentation issue and a fairly minor one at that. Please feel free to submit a PR to improve the documentation and thereby close the issue. It would be preferable not to derail this thread with minor off-topic issues – if you want to remind people about a GitHub issue, just bump the issue by posting a comment to it.

3 Likes

Seconding Stefan here, please comment to bump any issues you’d like considered for 1.0. I can’t guess which other issues you’re thinking of here. Note though that we’re mostly interested in breaking changes that could not be done in a 1.x release.

Sorry if my message was not clear, to make it clear: I totally agree with the approach you are taking, and would like to encourage the core developers to go on with the excellent work without being constrained by deadlines.

I understand that also the second part of my message was not clear (sorry again). I was referring to this proposal as an example, which was answered “This is an interesting discussion, but frankly, it’s way too late”. I wanted just to say that maybe some interesting proposals have been either discarded or postponed because they would have delayed 1.0 release, and maybe now they could be reconsidered. I reported it just as an example, and if as I now understand that proposal was actually discarded not because of time constrains, then it was not a good example.

2 Likes

Is there anything we - package developers and users - can help you with? In addition to migrating packages to Julia 0.7, of course.

4 Likes

It’s much appreciated – upgrading packages is, as you suggest, the main thing. If there’s some issue we may have forgotten about, bump it.

I was holding off looking seriously at 0.7 upgrades, since I was under the impression that it was still a pretty fast moving target (and because my package is still pretty under-developed, when I have time making it better has seemed higher priority).

I’m getting the impression here that 0.7 is now “close enough” that it’s worth reconsidering?

On a related note (ok to split this into a separate thread if that seems appropriate), is there an accepted best-practice (git-wise) for how to continue development on 0.6 while working on 0.7? Should I just have a 0.7 branch that I routinely rebase onto my development branch?

5 Likes

Yes, it’s very close now with a couple of caveats:

  1. If you implement custom iterable types, the iteration protocol is about to change, so be warned that you will need to update that too later.

  2. Some broadcasting behaviors may change as well, so if you have code that makes heavy use of broadcasting you may want to wait a bit. Then again, I don’t think the changes that we’re making will affect most typecial broadcasting code, it’s more about corner cases like “what if I broadcast a string?”

7 Likes

If you can, I think the easiest is to make your code work with both 0.6 and 0.7. This is often possible by a combination of using Compat.jl and conditional evaluation if VERSION >= v"0.7.0- ... of blocks which only work in one of them (ok to use eval if a 0.7 feature does not parse in 0.6).

But easier is to just freeze your 0.6 development and only add features on 0.7 from some point onwards. Then, if some critical bug happens, you can still do a bug-fix only on a 0.6 branch.

(yep, maybe this should be split off this thread.)

2 Likes

Thanks for the update! The work done so far ia stellar. Can’t wait to use heavily 0.7/1.0!

1 Like

A suggestion for migrating the ecosystem,

As always, make sure to bump versions such that the packages that depend on those packages are not affected and can tag new releases. My personal preference, many backbone packages are releasing Julia 0.7 only packages to take full advantage of the new improvements so it is worthwhile to decide if the package should take breaking changes that lead to better code or prioritize compatibility with 0.6.

4 Likes

I should say, thank you for taking the time to make this update. Keeping us at the sidelines in the loop is highly appreciated.

15 Likes

Thanks so much for updates and details! Very excited for the revision, and super in favor of delaying release to get this right.

Is this compiler tweak likely a “couple weeks” or a “couple months” project? I’m doing a Julia talk for the computing group at Vanderbilt in late April, and would like to offer some sense of when 1.0 (or rather, 0.7) may be coming out.

1 Like

We should have a betting pool, on when v0.7 is released (and how long afterwards v1.0 is released)!

My bet (and heartfelt desire, as I also don’t want to see the release rushed) is that it will be done sometime in July, and that by JuliaCon 2018, all the major packages will be working splendidly :crossed_fingers:

5 Likes

You meant, by JuliaCon 2019 right? Who’s working in July? :wink:

Anyway, it would be awesome!

1 Like