Did Julia community do something to improve its correctness?

General Usage
226

I mean, this is true of all software. And in fact the more people care about “correctness” the more bugs will be filed, so the one with the most correctness bugs filed may in fact be the most correct ecosystem. At least it’s could plausibly be the case.

3 Likes
227

I purposely did not go too far back in the tracker to pick those bugs out.

you could have made this argument in 2021, then encountered all those bugs in 2022
or made the same argument in 2020, then encountered all the lowering bugs fixed in 2021
etc…

I sometimes feel like I am being gaslit during these discussions (not just you — the topic has arisen multiple times). no matter how many examples I give, there is always some reason why “those don’t count”

does Julia have a correctness “problem” ? I do not have authority to say whether or not something is definitively and systematically a “problem” or not. It’s not enough of a problem to stop me from being excited about the language, its development, and to continue to use it for my personal projects

it IS enough of a problem for me to give pause about recommending it to my coworkers for actual production projects, and I can say that I have personally encountered more bugs in the language than I have with Python or C++

9 Likes
228

I’m definitely not trying to gaslight you nor minimize the importance of correctness bugs. All those bugs COUNT, I just don’t have the same experience you do in terms of hitting bugs or having a comparison to compare to and I don’t think there’s a clear data based reason to think that Julia is somehow worse than other languages. Obviously YMMV.

My main language before switching to Julia was R because I do mostly data analysis type tasks these days. And the fact is I just wouldn’t even try to do the stuff I do in Julia if I had only R.

10 Likes
229

This is ridiculous. Every single programming language has bugs. Looks at C++. There are thousands reported for most compilers. What would you expect? Perfection?

3 Likes
230

yes, I would expect to not encounter a single language bug during normal/vanilla usage. maybe that is an obnoxious expectation, idk. but it is indeed what my experience has been with Python and C++.

before you go linking issues to me, yes I’m sure those languages do have plenty of bugs themselves. but the crucial part is I have never encountered them. Maybe there is something about Julia that leads the user more frequently to edge cases.

4 Likes
231

Definitely. A lot of bugs seem to have been caused by the presence of OffsetArrays, something which may be possible in Python, but I’ve never heard of it. A lot of the blog post’s other issues arise because of mixing packages in ways that seemingly aren’t possible in other languages.(I think this was also discuseed somewhere above, but I don’t have the energy to search for it.)

7 Likes
232

FWIW, I have had the same experience as @adienes - I’ve worked with Python daily for 5 years and have never encountered a single bug in the language or the stdlib. I encounter at least 5 bugs in the language per year working with Julia.

The following ones are just the ones I’ve filed myself - and not regarding optimisation failures:

I’m also at the point where I do recommend Julia to colleagues with a caveat: I tell them that it still has rough edges, and feels like a language in beta.

13 Likes
233

probably because we don’t spend much CPU time in Python and its stdlib nearly as much as Julia? Basically anything we encounter as bug would be reported to one of Numpy/panda/scikit-*/TF/Keras/PyTorch/Jax/Numba/cupy… And we don’t try to compose our own complex stuff with vanilla Python nearly as much.

that seems unfair – if Julia is that bad (~in beta) we really shouldn’t be recommending them to scientists or companies, or really any production usage

5 Likes
234

just want to preempt blowing off all these issues linked by myself and @jakobnissen as inevitable challenges of complex composition

many of those are quite simple and arise with very plain code

4 Likes
235

probably because we don’t spend much CPU time in Python and its stdlib nearly as much as Julia?

I don’t think that’s right. When writing Python, I did do a lot of stuff in actual Python. It’s not like I just wrote 10-liners calling libraries. I certainly don’t have the feeling that I use “more” Julia now than I used Python back then

if Julia is that bad (~in beta) we really shouldn’t be recommending them to scientists or companies, or really any production usage

I wouldn’t recommend Julia to anyone who has strict criteria for correctness, no. However, for scientists, >99% of bugs in real life are not language bugs but ordinary user bugs, so having a buggy language like Julia is essentially a rounding error in the larger arithmetic of correctness problems.

2 Likes
236

exactly
if my code has a bug where the wrong value is being emitted (for whatever reason), we can lose a whole lotta money in a really short time…

1 Like
237

ok I guess it’s another round of issue # review then:

  • 51267 is edge case, “No, that is just the well-known issue that you are not permitted to both eval and execute code in a loop at top-level”. I don’t know what makes it well-known, but it’s definitely weird usage considering Julia already heavily discourage eval().
  • 47790, people can’t agree on if it’s really a bug, at most edge case
  • 46351, not a bug, just missed performance optimization
  • 43245, nesting stateful on itself, definitely a bug, but edge case to me (I was in the thread)
  • 43235, composition bug, me in 2022 thought it should just error
  • 40042, dump() displaying, not a bug per-se, needed triage to decide what to print
  • 38226, edge case, this is hand-writing LLVM? or at least, it’s ccall("llvm.x86.avx2.pmovmskb", ...)
5 Likes
238

now do these ones Did Julia community do something to improve its correctness? - #217 by adienes

239

To be honest, I don’t really appreciate you dismissing my entirely reasonable uses of Julia, and seemingly disregarding real bugs encountered in real situations.

Honestly, the uneasiness that you can encounter correctness bugs in Julia after doing completely ordinary things like calling cycle on a stateful iterator is hardly mitigated by someone claiming a year later that they think the code you wrote was an edge case, or somehow not really deserving of being considered a correctness bug. That just comes off as bizarre rules lawyering in order to not admit that Julia has a correctness problem.

Edit: It’s fair enough if you think these issues are not that bad. But it’s completely unreasonable to dismiss the bugs as not real bugs somehow.

12 Likes
240
  • 47410 bug
  • 51785, people have said about this, the else in try-catch-else is very new
  • 48889 bug but never got into released version?
  • 45162, performance issues are not correctness bug
  • 47168, (;a, b) is pretty new and I wasn’t expecting it to support const, TIL. Also not a bug, just inconvenience?
3 Likes
241

specifically to the cycle of stateful, I think we can agree it’s a composition Julia didn’t think about before. It should error because logically speaking there’s just no way to cycle a stateful without having a tape that records the entire history of the stateful iterator.

But for good or for bad, that’s how Julia ecosystem works – you try to compose and see if it works. I know you know this, I’m not trying to say you’re somehow not aware of this, but I think you may be wishing Julia ecosystem / convention is somehow different from what is in reality

We could have gone the other way – always lock down type signature and become more like C++, but we didn’t. And of course, we can start changing today!

I’m not saying Julia doesn’t have fair and sqaure correctness bugs, but I do want to provide some counterpoints / context if I see a big scary list of bugs.

6 Likes
242

What’s inevitably contentions about this discussion is that something like “Julia has correctness bugs” appears to imply that all applications may carry these correctness problems without knowing it. Then, people developing the enormous amount all the cool stuff that’s done in Julia feel somewhat dismissed as if they were ingenuous programmers incapable of recognizing the issues with the tool they use.

Personally, when I see the bugs reported here, what I wander (without any dismissing sentiment) is how people arrive to these things. How differently they code from what I do, to find these issues, and sometimes I learn something.

What makes me feel well is that most of those issues appears to be solved quite quickly. And what worries me is how many of them were fixed by one of the three or four language founders, which makes me worry about the community expertise to fix them in general.

14 Likes
243

We can go around pointing out particular bugs, issues, and pull requests all day. This is important and may lead to some resolution. Too often we need a thread like this to relist our issues.

Rather, we need to figure out a better social structure to help keep track of bugs, issues, and pull requests. This could be supported by infrastructure as well.

  1. Could we create a standing committee that works asynchronously to help manage and track pull requests? We have #triage, but I think this may be insufficient for the pull request volume we have.

  2. We have a Slack channel “#pkg-registration-manual-prs”. Could we have a Slack channel and Zulip stream analog for all “JuliaLang/julia” pull requests?

  3. Do we need to allow for more people to help tag pull requests in order to help sort and manage them?

10 Likes
244

The gcc compiler has 208 bugs reported in the last 7 days. Do you see anyone banging on about the “incorrectness” of C because of that?

7 Likes
245

No, but I see A LOT of people banging on about the incorrectness of C every single time some security vulnerability in critical IT infrastructure makes the rounds, because of course that critical piece of infrastructure is old & written in C.

Most often, those vulnerabilities are some use-after-free, out-of-bounds access or other overflow related vulnerability.

3 Likes