Did Julia community do something to improve its correctness?

General Usage
206

Well there’s General, we could add test/coverage/review requirements. That would be a systematic way to improve the ecosystem correctness.

207

No, that’s unworkable, ill-advised, and has been considered. I think you must mean the General registry, and does ANY open source ecosystem demand this? I’ll first quote, then explain, argue for and against:

Registered packages MUST have an Open Source Initiative approved license, clearly marked via the license file (see below for definition) in the package repository. [more similar]

  1. SHOULD be mentioned in the README file.
    […]

Registry maintenance

The General registry is a shared resource that belongs to the entire Julia community. Therefore, we welcome comments and suggestions from everyone in the Julia community. […]

Disclaimer

The General registry is open for everyone to register packages in. The General registry is not a curated list of Julia packages. In particular this means that:

  • packages included in the General registry are not reviewed/scrutinized;
  • packages included in the General registry are not “official” packages and not endorsed/approved by the JuliaLang organization;
  • the General registry and its maintainers are not responsible for the package code you install through the General registry – you are responsible for reviewing your code dependencies.

The rules seem very good as is. They are not “reviewed/scrutinized” since who is going to do it? You can do it, everyone can drop in with suggestions on packages after registration, help with adding test, documentation. Anyone is free to their opinion can “review” give their advice, but none can demand it followed, or block others from registration until such advice is followed for good reasons:

  • It feels like slavery. You are expecting unpaid work. Open source software is a gift to the community, that you can take or leave, and that includes Julia itself. You can however offer to help.
  • If you make demands, even one, conditions to registration, then we get fewer registrations! Do we really want that?
  • We could however have a star system (hmm, GitHub already does) or some kind of encouragement. Financial is not ruled out. I believe PyPi has systematic labels, alpha, beta,… mature. It might be a good idea, is it self-reported there?
  • Nothing rules out having a curated list of packages, and such exists already. Anyone can make their own, and anyone can make a local registry. Plausibly anyone could make a less-general registry with any requirement they want, i.e. curation. It’s been tried before when there was JuliaPro. Yes, it was proprietary (understandably, who should do curation for free), then it was I believe free or at least no-cost, then dropped.

But are there pros to requirements?

  • You would end up with better ecosystem. Maybe, or just smaller.
  • We already have duplicated packages, i.e. similar or same, for same concepts. Imagine if people were blocked from registering, we would have more such. Ok that feels like a con… So more unknown, non-public, that may or may not end up being registered. But we are keeping out those bad packages for the few curated reviewed ones. We already have two packages for object systems, since the author of the latter didn’t know of the former, which was registered. I’m not sure when work on both began, or this could have been prevented (they also have different types of OOP, so maybe ok). While I’m not proofing much regarding registered packages, imagine if more packages were developed in secreted, since not allowed to register.
  • Rick Hickey is against SemVer (which seemed like a good idea at the time, never break without telling with an updated version number), he wants packages to never break, explains how and why. making SemVer redundant. He believe when you develop in private you can break, and should. It’s up to you when you think you’re ready and decide to publish your software. But for open source, after that, if you want to respect your users, you can’t break ever. Is there a middle ground? You could register but label somehow, as WIP, alpha, etc. or 0.x which I believe is for exactly that. I’m not sure if he believes in only releasing 1.x, and never update that 1, or 0.x and staying at 0. something. I think he just doesn’t care too much about those version numbers.
  • He’s actually also against tests, or at least doesn’t consider them to helpful… Rather do as he says, not place-oriented programming (i.e. immutable) and get rid of bad software that way and breaking changes.

We could add JET.jl and

or whatever tools to the General README, as hints on what you can do to help with development/correctness, and advice that they be used prior to registration and after. We demand that package are not empty, or near-empty, i.e. helpful to someone, even yourself. A very low burden. I think we can help with offering tools, such as CI, suggest them used, but I’m conflicted on requiring even one test. Some do not believe in tests. Those that do can ignore packages with no or few tests. Nobody is forcing you to use a package. If people are ignorant of test-driven development, or simply against, then documentation of it is ok, to help people learn and encourage it used.

People may be beginners when making their first package, and people hopefully learn. Over time packages can get better, also with help of others. Should there be requirements going forward, or to upgrade packages? I think the same applies. We currently want people to signal breakage with SemVer, at least be aware people are breaking packages, hope they do not, or if signal it. But is it a requirement? I hope you’re not hiding breaking changes from people intentionally. Even forcing people to upgrade (or not) the major number seems too much to ask, only something we can do politely.

1 Like
208

I know Conda does, and there may be other examples.

209

As of today, both are offered as options by PkgTemplates.jl on the master branch, and soon in the next release :slight_smile: I’ll draft a PR to suggest them in General

EDIT: here’s the PR

7 Likes
210

I was curious about this so I looked into it. It’s important to be specific here since the whole Python package system is very confusing (at least to me).

Conda is an open-source package manager, not a package repo. Some Conda repos are “are built, reviewed and maintained by Anaconda®.” So, the repos may be open-source, but the heavy lifting is done by a for-profit company by people who (presumably) get paid for it.

Conda-forge is an open-source, community-led repo where “[the] review team will assist you by pointing out improvements and answering questions.” On the conda-forge mainpage there is a list of supporters, both financial and for infrastructure. So, the “community-led” repo is still backed with money and people.

The point has been made before here, but its worth reiterating. People can only volunteer so much of their free time to assist open source projects. Reviewing and assisting every new package and author takes time and money.

Also, I could not find anywhere the actual guidelines for publishing to the repos. So while they are “reviewed”, I have no idea what criteria are used, and without that I don’t think it can be claimed that a random package from a conda repo is better than a random package from the General registry.

Edit: Imagine if the General Registry reviewed and blocked packages according to some unpublished rules. People would be pretty steamed.

4 Likes
211

I believe the original blog post by Yuri has made a good attempt at explaining what “correctness issue” mean in the context of our discussion (and given there were 3 different discussions already on the internet either in or out of Julia community, one could gain a good idea of what “correctness” in question is by reading through the discussions on Reddit and Here).

But yes, I definitely agree with a bit more documentation.

212

Yuri has made substantial contribution the Julia Language and his point on “mathematically correct” was, to the least, mostly valid; the post is here, feel free to give it go. Another community member later in this discussion has confirmed (just a while back) that there were 25 issues with “correctness”; Yuri alone has raised quite a few.

The possible middle ground here, is that it’s almost 2 years on since Yuri’s post and Julia certainly has progressed; secondly Yuri has raised some edge cases for mathematical correctness which for the majority of us, are just ok to live with.

1 Like
213

there have unfortunately also been a couple bugs—even recently—in lowering, and these kinds of bugs (to me) are much more painful than the map! variety of bugs. when something like an if-else block or the scope of variables is not behaving as expected, it makes writing code feel like tapdancing on a tightrope

2 Likes
214

Great point; we can start with whether array aliasing is something we’d allow in sum! and likely the conclusion of this could warrant us some warning in the documentation either way.

2 Likes
215

Could you provide links to the bug reports on GitHub?

216

[I was also curious to look that up, but since already done, I just took a quick look.]

I can’t confirm what Conda does, except select, i.e. curate packages. Something I explained is highly possible externally, or some flag could be added to General that states approved package.

I.e. all go in of varying quality, and some are better and marked (but not denied registration, just the mark, until approved). What could be that mark? One condition (or only condition) could be at SemVer 1.x or higher (we could have community-maintained labels of packages; if Chris, either one, has used and approves of the package, I would too), Anyone can actually just choose to NOT install 0.x packages, already; maybe the Pkg should warn, or refuse, for such packages? I suppose it can’t now, I don’t think it would though be a breaking change. It could warn, and we could consider denying later as default, and then people could configure either way.

It’s a question, should a package then at 1.x, but depending on 0.x be denied too, or warned about? It could be argued that the maker of that package has preapproved its dependencies (or not…). It could be a setting to be even more strict and deny recursively.

I see:

  • Over 250 packages are automatically installed with Anaconda.
  • Over 7,500 additional open-source packages (including R) can be individually installed from the Anaconda repository with the conda install command.

So the curated list of packages is pretty small, or should I say 250 is tiny? 7750 is also not to large, smaller than 10,000+ of Julia. Those are not comparable numbers for Python, if you should exclude R packages.

I think it also has a much larger list, but even that complete list in conda, is much smaller than in PyPi. Now the question is, are all the packages in conda good, since preselected? I know people object to conda since they are missing some (presumably) good packages you can install with pip.

[I’m not sure I think pip installs from PyPi only (contains all registered Python packages; and only, i.e. no R), conda includes mostly a subset of Python, also some R, not much else, though a few Julia). I and others thought Conda.jl can also install from conda, but FYI, I’m told it can also pip install, so PythonCall.jl has access to all Python packages I believe.]

1 Like
217
5 Likes
218

Not sure if you’re just trolling anyway, but “mathematically correct” doesn’t appear on that page.

219

no, but “incorrect mathematical” does

220

Just an FYI, this does happen. Usually the package author ends up agreeing that it shouldn’t be published to the general registry after people independently go in and take a look at a package, but not always.

Ultimately, the general registry has very few rules and is very permissive, but it’s also not a garbage dump, and people do put some level of effort into gatekeeping packages they think are useless even if they technically meet the official guidelines.

4 Likes
221

True, but there’s no discussion of mathematical correctness nowhere near, the paragraph is about @inbounds misuse.

I definitely agree that a somewhat careless attitude is/was present in some parts of the ecosystem regarding using dangerous features like @inbounds, @pure, etc.; and the devs can definitely be blamed for this somewhat, seeing as they could have, e.g., included unsafe as a prefix to these names.

Also, I’m not the kind of Julia fan that thinks the Julia devs or community can do no wrong, for example I was very recently disturbed by some instance of rude behavior directed by some Julia devs towards another on public Github; and I have criticized Julia publicly myself.

However this whole specific thing of attacking Julia (either the language or community or both, usually it’s unspecified which one) is IMO weird, way overblown and mostly based on vague and misleading claims and loaded language. Someone seems to be pushing and mysticizing the “correctness bug” phrase to the degree that it now seems to be largely associated with the Julia language (searching “correctness bugs” on Google yields Julia-related results on the front-page, even when I try with “Incognito mode”, FWIW). I find the singling out of Julia here to be absurd and suspect. I’ve seen many HN comments associating “correctness bugs” with Julia, and I notice the comments are usually dumb, e.g. they don’t even specify what they mean by “correctness bug”, even though the term is obviously neither well-defined nor common. Also, they always point out the same, but comparatively benign bugs, indicating a possible lack of familiarity with Julia. I suppose there was some sort of troll/sockpuppet/keyboard army campaign of discrediting Julia, for whatever reason.

13 Likes
222

how about the 5 I linked above then. wouldn’t call them all benign

223

I think “edge cases” is important because they’re only going to show when someone finally touches that edge. I gave the article another look (I read it when it first came out) and found that almost all of the bugs reported in his initial two lists where closed and merged (yay!). Though I didn’t check them all by any stretch.

Just for comparison, NumPy has nearly 2000 open issues, 634 of which are labeled “bug”. Granted, numpy is a huge project and one would expect more issues than a small Julia package. But it still has 10 full pages of bugs older than Julia v1.0 (~Aug 2018). SciPy has 440 “defect” bugs. Do NumPy and SciPy need to do something to “improve their correctness”? No, and I have never personally encountered a bug in NumPy or SciPy.

I don’t mean to pile on, but much like @nsajko I get the sense the folks are losing the forest for the trees.

7 Likes
224

Thanks for the links.

47410 was fixed almost a year ago, and requires quite specific (and I’d say uncommon) syntax to trigger.

51785 has been merged; this should definitely been caught by testing.

48889 has been fixed; should have been caught by testing.

47168 was fixed over a year ago; should have been caught by testing.

45162 was fixed over a year ago; it does not involve any incorrect results.

I’d prefer not to see these bugs occur, but hardly make me feel like I’m dancing on a tightrope.

If anything, I’d say that what is missing is enough tests to cover all uses of new syntax as it is introduced. This is made especially difficult by the flexibility of the language.

3 Likes
225

yes all of those have been fixed now because they are the ones that were caught. I cannot provide examples of bugs that I don’t know about yet. but the point is that I think every one of these should have been caught by testing

does not exactly inspire confidence as to the robustness of lowering logic. in fact I have encountered issues myself I’m pretty sure are bugs which involve putting too much logic inside default function arguments. next time I can recreate it will file another issue

2 Likes