Known bugs in new Julia releases

Internals & Design
,
1

Before we release Julia 1.8 I would like to get some
clarity about how severe a known bug must be before
it needs to be fixed before the next new release of Julia.

Two months ago I reported what I thought was a bug
that was important enough to be fixed, but it seems that
no real effort has been put into fixing it.

It is a bug that crashes Julia and is easy to reproduce with a 10 line
example that uses no external libraries. To me this seems like enough
to warrant a serious crack at fixing it. Was this an oversight, i.e. should this have been tackled
before rc1 was released? If not, how serious must a bug be before it
is considered worthy of a serious attempt at fixing it?

6 Likes
3

AFIAK it is used for researching where correctness is mandatory

4

The bug mentioned by the OP is neither a correctness nor a composability bug (and it is also not about the ecosystem).

5

Looking at your issue I see that at least two people have investigated it, one of which even git dissected the issue to find the relevant commit that introduced this. I would think that the main reasons for why this hasn’t been fixed yet are

  • the relevant code is in julia-syntax.scm, i.e. deep down the rabbit hole and written in Scheme. (limits the number of people who can realistically tackle this issue)
  • Jeff, who introduced this, is a very busy person.
  • Presumably, the bug isn’t the “most important” of all the bugs out there, i.e. most people will never hit it.
  • (Fixing bugs certainly isn’t the most rewarding and fun task. Typically you need to pay people for this kind of work.)
5 Likes
6

Thank you for your explanation. While I do realise that many like implementing new features more than fixing bugs, as a user I would prefer if stability and correctness were given a higher priority. To me, Julia is already light years ahead of its closest rivals regarding language syntax and execution speed. I would prefer to have more bugs fixed as that would give me more confidence in the overall stability and correctness of Julia. When there are many small bugs like this one it adds up and becomes more likely that I will encounter one. Also, if bugs are not fixed it discourages people from reporting them and more serious bugs might go unreported. Finally, sometimes the process of fixing one bug triggers the developer to discover a more serious unknown bug.

So I do request the Julia development team to consider this.

10 Likes
7

Which language do you have in mind?

8

I am talking about Matlab and Python and true, this of course applies to the code that I write and not necessarily what others write.

9

Interestingly, no error is produced if I follow Julia’s performance tips and wrap captured variables in let blocks when defining a closure.

Original code which crashes all supported versions of Julia:

begin
    local x::Tuple{<:Any} = (0,)

    f = () -> begin
        x = (1,)
        return x
        
        size(x) # must use x
    end

    f()
end

“Fixed” code with a let block:

begin
    local x::Tuple{<:Any} = (0,)

    f = let x=x
        () -> begin
            x = (1,)
            return x
        
            size(x) # must use x
        end
    end

    f()
end
1 Like
10

Julia is still really a community driven project. We do not have a large tech company with nearly infinite resources developing Julia. An important aspect about the bug you found is that it has a work around.

As others have mentioned, the bug appears to be in a relatively esoteric part of Julia, the parser, that is written in FemtoLISP. There’s really only a single person that fully understands it.

One idea is that perhaps we should rewrite the parser in Julia itself. This is what @c42f has done with JuliaSyntax.jl:

Given limited time and resources, we could try to fix the bug you found or we could spend that time switching to a parser that is less esoteric. At the moment, I am very curious if JuliaSyntax.jl suffers from the same bug. Understanding that might actually help us figure out how to solve the original bug that you reported.

5 Likes
11

The good news [EDIT: it may fix some bugs, see below, but not that one, or will it, still worth a try?] is that the Scheme (FemtoLisp) code is totally replaceable, already done with JuliaSyntax.jl, so you can check with it. There’s a PR about adding it into Julia, but users don’t need to wait for that.

Otherwise the last known version to work is “Version 1.6.0-rc1.19 (2021-02-16) 56a3fb7” and also 1.5.4 can be used as a workaround.

how serious must a bug be before it is considered worthy of a serious attempt at fixing it?

All bugs are serious, some more than others, e.g. (security bugs or) bugs introduced since the last supported Julia version (or LTS).

I didn’t analyze much, but there are 222 open bug (some very obscure), thereof 5 in the parser (would be curious how many JuliaSyntax.jl fixes, or possible adds…), the with the least recent commentated on/by:

“JeffBezanson added bug parser labels on 4 May 2015”

The oldest, from 13 Jun 2013:

I checked and GCC has 500 bugs in:

  • Component: c
    This result was limited to 500 bugs.

same with Fortran there, and more components are missing for that C compiler I guess, and GCC C++ compiler, and its libstdc++ component is the same story.

D component of GCC (the separate language D, it also has a different compiler):

242 bugs found.

To be fair LLVM has “138 Open bugs” and I guess you must add (all?) of them to Julia too.

I think the moral of the story/investigation is compilers have bugs…

12

The bug here is in lowering or codegen (yes, the change that triggered it was in julia-syntax.scm, but it could be that the frontend is emitting a valid-but-previously-unexplored AST that codegen can’t handle), not parsing. JuliaSyntax.jl only covers parsing, though its goal is to cover lowering eventually, too. There are more folks than Jeff that can read and write the lisp code, but it’s true it’s a small subset of Julia devs.

In any case, this is all quite off-topic from the question at hand. It doesn’t really matter what language is used here — it’s a bug and should get fixed. It just takes time, effort, and attention.

12 Likes
13

First of all let me say that I made this post to find out if bug handling in general in Julia needs to be improved, I am just using the bug that I found as a concrete example. I realise that resources are limited, so there are two options, one is if we can motivate the community to fix more bugs and the other is to reallocate resources from development of new features to bug fixing.

My impression is that the main problem is that the core team behind Julia is not taking bug fixing seriously enough and that this is causing the community to either focus less on bug fixing (or use Julia less). My impression is also that a even a small reallocation would make a big difference. If it is such that at present 10% of developer time is used for fixing bugs in released versions of Julia and 90% is used in developing new features and fixing pre-release bugs, then reallocating to 11% vs. 89% would be a huge improvement. If we all walked around proud that “our” Julia has almost no bugs and that it is one of the most stable compilers around I think that this would motivate the community to do more bug fixing as well (and it would be good for getting more people to use Julia). I think that an attitude shift like this from the core team would spread to the community and would be very beneficial for Julia.

(I tried JuliaSyntax but it did not prevent Julia from crashing due to this bug)

14

In free software, users do not get to drive.
My experience as a maintainer of other projects (I’m just a user here),
is that I work more efficiently when reports are factual,
and users give ideas, but otherwise let me drive in peace.

Your bug report was very good and useful;
please consider that it might be very difficult to solve,
and that fixing this might require some deep changes, taking time to keep the overall quality.

90% is used in developing new features

I don’t know where this figure come from, but given the overall quality of julia,
there is a chance that it is grossly exaggerated ?

A possible perception bias: the NEWS indicate new features and changes, not bug fixes
which are already known by people who followed the issues.

5 Likes
15

I don’t know how easy it’s to fix your bug (at least it’s being looked at), but I know it’s a regression, and there are currently only 5 outstanding regressions (thereof one on the 1.8 milestone), and 255 fixed:

A bug has to be fixed then it gets into the next release, or be put on a milestone to delay a release. I see a bug with a similar pattern as yours:

Why was this added to the 1.8 milestone?

Showing the wrong answer seems quite a bad regression

Yes, but not vs 1.7. If all supported Julia versions are released with this bug, we surely can release 1.8 with it as well.

@JeffBezanson JeffBezanson removed this from the 1.8 milestone 9 days ago

16

For the reasons Matt explained this cannot help, unfortunately. It’s an issue later on in the compilation pipeline. I intend to make JuliaSyntax.jl do lowering (or parts of it) in the future but we’re focusing on robustness, improved diagnostics and Base integration for now.

Regarding affecting allocation of development resources in free software - I believe the only effective and immediate way to do this is by allocating your own resources toward areas you care about. If that’s bug fixing, it’s by writing good bug reports and/or diving into the code to fix them, or paying someone to do that on your behalf. Conversely, it’s hard to convince other people to allocate their own scarce/valuable resources to specific areas of the project when they’ve already got a pile of things needing their attention.

Such a situation may not feel ideal or fair; in particular it can focus attention on the needs of users who also happen to be developers (ie, underprioritizes needs of parts of the user base, especially beginners). But on the other hand, developers should be free to decide where to spend the scarce resources that belong to them. Overall I feel like it works out fairly well as a distributed mechanism for prioritization. In many ways superior to commercial - especially enterprise - software where it’s common for none of the direct users get a say in allocating development resources!

7 Likes