Did Julia community do something to improve its correctness?

Just searching the string “incorrect gradient” in the issues of pytorch, jax, and tensorflow, apparently yes, some silent, some not. However, I’ve never done ML so I can’t say how important or prevalent these issues are, let alone make comparisons to Zygote.jl’s. I can only take the word of people who have used all these tools thoroughly in their work.

It’d be very strange to block a release that fixes some issues and adds demanded features just because other issues and features aren’t resolved yet. You mentioned Numpy’s outstanding issues, obviously releases kept coming. Developers need to agree to block a release to fix an issue or include a feature, it’s not a default decision.

Like I said before, correctness isn’t One Thing, some nebulous force threatening the future of the language. It’s just a fact of life that software has bugs that get patched incrementally and routinely. It’s so mundane that every big feature had to resolve their fair share of correctness issues at some point, there’s even a state of Julia slide about the progress on threading that casually lists “features/correctness” topics (each topic can have multiple issues).

5 Likes

Legitimate question: are Julia’s bugs seen as of a different character than other peer projects? E.g. the same kinds of issues seem to be present in other projects, e.g.:

E.g just some cursory searching finds incorrect results when using modulo arithmetic in rust or odd edge cases in Numpy related to mutability, complex-number types, unsigned integer behavior.

To my uninformed eye, these problems seem similar to those that have been identified in Julia?

Also, at the time of writing Julia has fewer open issues matching “bug incorrect” than Numpy does, though I admit not enough understanding of the nuances here to comment whether that metric is meaningful.

5 Likes

I think the nature of coding in Julia leads one to encounter “edge” cases more frequently than other languages, so it is more painful when these are wrong.

As an evocative but probably not-quite-right example, in Python if you have object types A and B and you want them to interact, one might first transform them into a very safe/robust/well-tested form, like a vector of floats or a dictionary etc., then perform the needed operations.

In Julia, one might call foo(::A, ::B) directly and just hope that each type is satisfying its interface well enough for foo to remain correct without any transformations to a “safe” type needed at all

8 Likes

Every single open-source project I know is basically starved for personpower. We’re volunteers too, and while we do occasionally beg for contributions, you shouldn’t need us to do that in order to realize you can help.

That doesn’t mean your PR will be reviewed instantly because of the issues pointed out so nicely in Did Julia community do something to improve its correctness? - #103 by gdalle, but hopefully someone will do it justice eventually.

11 Likes

It didn’t cross my mind to link the begging for contributions to being able to assess that I can help.

I am not sure to what extent others feel the same, but when it comes to evaluating what one can do, I am pretty confident when doing that evaluation for the projects I am working on as opposed to contributing to Julia-repo.

I know that there is always the make a PR, and let us tell you if it is garbage solution - but that feels more like fighting my way in approach. The same “…we’re volunteers too” applies here - and both the following were valid for a while now:

  • I really love Julia and I want it to thrive
  • I have (can make) time to contribute

In fact, I wanted to contribute in some way and after this topic cooled down I started to be more active here, on discourse.

I know that general true statements about what contributing to open-source projects means can be invoked and applied to Julia (and it is not like I am ignorant about those rules): however, I think we can at least entertain the idea that in a less technical way than, now mitigated, TTFX, Julia might have a time-to-first-PR issue (and is less relevant that other open-source projects might have this issue as well).

It was a clear TTFPR issue when I reported the following bug after I actually delved into Julia’s source code, detected the exact issue, and proposed the fix.

So why didn’t I PR the thing and only open an issue? Good question: I cannot pinpoint a single reason.

Maybe it was about a mix of various factors. But my honest answer is that I wanted to contribute, I had the time (and actually I had the local fix because the bug was really annoying) and still no PR.

If my case is singular and this I want + I can + I don’t mix is not a real issue that goes past my sole experience, then I think we can just let this go: I learned my lesson.

However, if there are things that can be done in the Julia community to reduce yet another TTF- something, then let’s use this opportunity and maybe lower the perceived bar for Julia-repo contributions.

3 Likes

I know that many people don’t contribute who perhaps should, but also that many who do submit PRs can have a frustrating experience waiting for reviews. I don’t know how to fix that. The fundamental tradeoff is that time spent reviewing PRs (and it takes a lot of time) means time not spent on other activities like fixing really hard problems. So we persist in an awkward balance where good stuff gets done but perhaps more slowly than ideal.

Anyway, this is a bit off-topic from this main thread, I’ll stop now. If you have ideas about how to fix it perhaps another topic?

7 Likes

I agree we should stop here for now. I’ll think about this.

Also - thank you for the hard work you put into the Julia ecosystem. Sometimes it is hard to point out some issues without sounding ungrateful/demanding: so again, thank you!

4 Likes

So here is a many page discussion about the reverse-zip bug I was about to fix, but when I saw from the lack of enthusiasm that the fix would mean having a PR lingering around for years again I left it be Iterators.reverse gives unexpected results when zipping iterators of different lengths · Issue #25583 · JuliaLang/julia · GitHub (I had already with #24978 a PR that took 3 years and with #29927 one that took 2 years)

1 Like

If you’re interested in discussing this specifically I’d recommend starting a new topic or reading over some of the previous discussions, but yes this specific example is probably the poster child of the hype and shiny features over correctness culture being decried.

However (and note this is coming from someone who has complained at length about the aforementioned issues), it does feel like one of the more extreme examples and not representative of the ecosystem as a whole. Case in point, ForwardDiff.jl has been quite solid for years now in the AD space. The reason this is a big deal despite only involving 2-3 libraries comes down to a) interest in ML as shown by anecdata and successive Julia Community Surveys, b) some of the packages being promoted as flagship libraries in the past such that everyone and their dog knows about them, and c) excessive hype about the capabilities of this specific corner of the ecosystem during early development which didn’t pan out for a variety of reasons.

I think you’re right to be skeptical about number of issues being a good metric here. Suffice it to say Zygote has/had a greater number of significant issues and issues in commonly-used code paths than the Python ADs being compared against, even if the absolute or relative number of issues is similar. It does seem like some lessons were learned from this, but unfortunately not before the Julia ecosystem experienced a reverse-mode AD “winter” which has only recently begun to thaw (again, a discussion for another thread).

5 Likes

It seems this thread has come to an awkward end. Can I take credit for that?

3 Likes

See What’s the aliasing story in Julia

4 Likes

(Edited) It was fixed after beta1. Still tagged for backporting. It’s been merged into Backports for julia v1.10.0-beta2 by KristofferC · Pull Request #50708 · JuliaLang/julia · GitHub so it should be in beta2.

7 Likes

Recently I’ve got a project that’s big enough for a deployment and also fairly technical, I was torn between Rust (Rust isn’t built for technical computing but the ecosystem is catching up fast and other advantages provided by Rust are almost too good) and Julia (I always wanted to try Julia for some real work in science computing, whereas I’ve been using R mainly and MATLAB several years back); but my attempt to do some work in Julia has come to a halt mainly because of Yuri’s blogpost. I’ve followed the Reddit’s discussion on Yuris’ post, and an earlier Julia community discussion on the same post.

There are, by now, a good number of opinions exchanged between both sides, with regards to the correctness issue raisd by Yuri in all 3 discussions. But I would really appreciate a formal address from Julia Dev Team on the correctness issue in Julia, regardless that the team agree or disagree (Or probably there is one already, please kindly point me to it).

Some argued that we should let Julia mature and these types of bugs were so common that many languages suffered before and time would cure all. Opponent disagreed with examples of R and Matlab Core. The Core R and Matlab really has been robust for very long (abundant evidence from all 3 discussions and other places). CRAN (R official package management), is too pedantic (almost notoriously) such that the majority of graduate students would not even consider publishing packages.

Some said that it is the third party packages from researchers and graduate students who were being a bit careless when it comes to programming (rather than solving math problems using computer language). Yuri however brought attention to Base Julia near the very top of the post and they were raised in Julia 1.7 ish back in 2021 or 2020 (e.g., prod function from Base, raised in Jan 2021, Yuri’s post was written at least before May 2022). It’s the number of correctness issues Yuri found and the likelihood of similar issues still pertaining in Core Julia that worried Yuri, in the first place at least. I believe OP was worried much, henceforth asking about new measures for correctness being employed or not. This is the very thing I’d hope the team would address as well, as a potential Julia user.

Others mentioned testing suites available to users such as Aqua.jl or JET.jl; I am not sure I, as an user, ought to write unit test on Base Julia functions as singular as prod, or to set up tests only to almost frequently find out that it was the Base Julia I’d have to fix.

While most critical posts of Julia from Google were merely complaints about user experience and how hard they’d have to depart from old habits built in other languages, Yuri’s post, on the other hand, constructed valid points on one of the main objectives of a technical computing language IMHO; that is to be correct in computation result.

I like Julia since it came out (I’ve been playing around in bitesize since 0.8 but I’ve not really used Julia for anything); I appreciate and support Julia’s ambitions, be it to solve “two-language problem” or to offer syntax similar to MATLAB with performance almost on par with C/C++; and yeah, multiple dispatch as well. Hand on heart I’d reall hope Julia to thrive and be the future of technical computing.

5 Likes

while it’s hard to quantify for sure, but I figure someone could write a Yuri style blog by picking oldest things out of: https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AI-unsound

the fact there are bug reports is a good thing because it means people are using it. But if you look at Rust or Julia’s list of “correctness” bugs, some are very-edge case, some are arguably correct just surprising. I think Julia itself is not more buggy than any major programming language kind project.

9 Likes

Could you define what you are looking for exactly?

  1. Who is the “Julia Dev Team”?
  2. What constitutes a “formal address”?
4 Likes

There’s not been a formal address from the project (it’s not really clear what such an address would look like), but many prominent contributors have indeed weighed in on the many discussions around these — in addition to the Reddit and prior discourse threads, see also the Hacker News discussion from when the post was first published.

I see the hardest issues in Yuri’s post being symptomatic of — as Stefan writes in the HN thread — “the flip side of Julia’s composability is that composing generic code with types that implement abstractions can easily expose bugs when the caller and the callee don’t agree on exactly what the abstraction is.” And that’s where things get interesting. There’s some really cool work to formalize what those abstractions are (e.g., [ANN] RequiredInterfaces.jl). But as I myself previously said,

Yuri really pushed on the ecosystem — and indeed most of the issues he listed are about the composability of packages. As someone who sporadically interfaced with him on-and-off in issues like these, I earnestly thought he enjoyed living on the cutting edge and slaying these dragons. That’s really the biggest loss in my view: that he burned out doing so and we lost his voice and work from the community.

So: Are there still dragons out there? Most definitely — especially if you find yourself bushwhacking through the weeds of packages nobody has tried together before. But it’s continuing to be easier and easier to stay on golden brick roads as Julia and its package ecosystem continue to develop and thrive.

33 Likes
  1. I am not sure who’d be the Julia Dev Team in this case; off the top of my head it probably could be someone, or some group who has had a focus on the development in relevant areas, who can taken ownship? Or probably from the authors/founders of Julia? I understand contributors to Julia Core have participated into the discussions of Yuri’s post.

  2. But isn’t this “community” discussion? community discussion for me surely isn’t formal. By “formal” I guess I’m inclined to imply that I can be certain there is/isn’t future effort or internal discussion from some contributors/subgroups who own the relevant areas (I may well have missed them and I’d appreciate if you could point me to them). As mentioned by @mschauer, this thread has come to an awkward end. What do we know for sure that can answer OP’s question, out of this thread? Probably the contributors who’ve participated and cared before, could come up with some agreed response to the community?

yep may well be; Yuri brought our attention to correctness and I agree that a good number of them were edge cases; there were not so edgy cases as well that’s what worries him I guess. I admit that I probably fret too much having reading Yuri’s post given how much I’d want to use Julia in the new project; it is the very intention that I wanted to try Julia that stopped me here when correctness has very high stake in a project. Yuri’s experience together with other public accounts, have really done a good job in validating their point.

1 Like

I recommend you give Julia an earnest try for yourself :slight_smile:

4 Likes