Improving the Julia issue tracker

Meta Discussion
1

Hello!

This post has (in part) come to be due to a general feeling of the issue tracker for julia being a little hard to work with as a non-core developer or absolute beginner, wishing to contribute. The goal of this post is to

  • Identify problems users have with the way the current issue tracker functions
  • Figure out how we can improve the experience for contributors
  • Fix more issues that people are running hitting, that may languish for a while

At the moment, opening an issue presents us with the following view:

screenshot_2022-06-26_19-46-22_104477052
screenshot_2022-06-26_19-46-22_1044770521132Ă—559 83.4 KB

This is, in my opinion, quite bad. It’s a wall of text that few people seem to read (judging by the number of new issues created that don’t have a versioninfo() attached) and those who do can only get the bare minimum out of it. Since there’s no distinct format for new issues, it’s also hard for newcomers that maybe want to help out to figure out what the problem (if any) even is. Further, new issues by default are not labelled at all, leading to loads and loads of issues that are just never assigned any label at all (at the moment, 1056 open issues exist that have no label at all, 146 of which were opened since the release of 1.7.2).

What’s worse though is that of those issues that do have a label, most seem to just… sit there. Some get someone who reproduces the issue or get some acknowledgement, some are fixed with PRs - but over all the majority of new issues that are not trivially closeable (mostly due to belonging on discourse in the first place - another point towards the “wall of text” being a bad first hit) just seem to languish. I don’t have numbers for this, but looking through the monthly repo insights and looking at the issues that are closed vs. those that are opened, as well as the PRs that get merged, certainly paints a picture of only the most critical things being fixed.

One way this could be improved in part would be to make use of issue templates:

(I hope that link works.)

As an additional anecdote, my personal experience has been a mixed bag of diving into library functions, trying to fix an issue I encountered, only to discover that I have no clue what the code is doing and no comments and noone there to help out with what it’s supposed to do. I haven’t counted, but the number of times I got some variation of “I don’t remember” that could have been solved by some internal comment/documentation is… too high.

Another problem that I’ve personally encountered multiple times is that PRs that are not related to critical, release blocking bugs or new features just… sit there, for weeks, sometimes months at a time, even if they’re QOL related, have green CI (which is surprisingly rare, spurious failures seem to be the norm), have NEWS.md and are really ready to be merged. This is very discouraging and takes the wind out of wanting to contribute, to say the least.

I think this could be improved by actually continuing to focus on user reported bugs, issues and QOL improvements. Feature-wise, julia is really great, but QOL-wise not so much… Mind you, I’m talking about Base exclusively here (though I’m certain parts of this would also apply to larger parts of the ecosystem… perhaps this can serve as a “food for thought” post to organize the workflows of packages a bit as well?)

I have no clear solution for PRs sitting around, other than “Please merge them?” and I’m not sure how to make it easier for newcomers to create their first PR and/or get guided into making better PRs, other than trying to get core devs to also try to improve their PR/code game by proactively writing more internal documentation. I know that it’s boring, but I feel like it’s gotten more and more necessary with the size the julia project has reached now.

–

What are your thoughts about this matter? Do you think there even is a problem? What do you feel about contributing more? What could be done better from core devs right now? What would make your life easier when navigating the issue tracker & contributing?

Anyway - I hope this can stir some discussion about how we can improve the state of things. Please keep it civil in here! I really don’t want this to devolve into a flame fest.

16 Likes
2

I feel like maybe we could use bulleted list and use of better wording(I am not good at that) coudl help:

  • If you submit a bug: Provide a minimum reproducible example, and if necessary attach versioninfo()
  • If you experience problem with a package: …

Other texts I find not easy to shorten.

Also versioninfo could be easily useless information, sometimes only julia version is enough, so don’t force it, while asking for more details from devs could be a good idea on initial post.

1 Like
3

It would be nice to have an issue tracker which asks whether what we have is a bug or a feature request or something else. And then provide a template based on that. Some examples:

  • Rust: Has a lot of categories which might not be relevant for julia.
  • LAMMPS: Has fewer categories and seems one which we can follow.
2 Likes
4

Having more people triaging issues would also help. We get great help from some people but we could use more to reduce burden on more senior developers by being a front-line of reviewers for issues & PRs.

I agree with the frustration of waiting for reviews or merges but I also understand exactly where it comes from. For example, in the time I have available for Julia I’m working on reducing TTFX, probably the most-hated aspect of Julia’s usability. Should I set that aside to review PRs in a half-dozen areas that I know well (but am slightly bored by), or should I prioritize TTFX work? There usually isn’t time for both. Likewise, other people put enormous effort into compiler correctness & efficiency; only a few people can do that, and every hour they spend doing something else is slower progress on something more fundamental. The reason you see prioritization of certain critical bugs and specific features is that the people doing the work have identified those as the subset of things they have time and expertise to work on. For some people, “expertise” is the limiting factor, but for others it’s 100% about having enough time.

I know that submitting a PR and have it languish disincentivizes future contributions, so that’s not great either. It’s an important issue but there’s no easy solution short of having a larger number of people take responsibility for specific systems within Julia.

14 Likes
5

Something that I’ve thought about doing (but never found the time for) is making a little script for building the latest Julia master with a list of PRs merged in. These builds would be even more “bleeding edge” than the regular nightlies, giving quick and easy access to features/bugfixes in PRs.

Hopefully one could make an interface that is easy to use and doesn’t require detailed knowledge about git and build tools by the end user.

This could also be done “in the cloud” (e.g. using Github actions) making builds from user-specified lists of PRs. These lists of desired PR:s could then be used to prioritize review and merging.

1 Like
6

Concerning help with triage, adgjlsfhk1 on Hacker News mentioned the core devs are liberal with giving permissions to new helpers on the triage front. Is this something the core devs could elaborate on? I am an average coder and have already created some good quality julia libraries, but I have not participated in many Julia github conversations, so it is not clear to me whether I would be taken seriously if I try to help with triage. And having the triage “Standard Operating Procedures” written up somewhere would help with newbie volunteers like me.

3 Likes
7

I am adgjlsfhk1 (I had a hacker news account with my normal username, but I set it up as a throwaway and lost the login info). In general, if someone has merged a PR (or just meaningfully contributed to discussion on several issues), I would consider them eligible for the triage role. It doesn’t allow anything scary, so the main decider is if someone bothers to click the buttons needed to give out the role.

5 Likes
8

Procedure for granting triage roles

We may establish a procedure.

  • Create triage role groups in the JuliaLang organization and allow users in the role groups to add tags to issues.
    Maybe there’s already one. @viralbshah
  • Set some parallel conditions for deciding whether someone can be granted a “triage role”
    • e.g.: (has merged a PR) || (meaningfully contributed to discussion) || ....
    • We may discuss it in this thread, or start a new thread / issue
  • These conditions need to be listed in the right place, so that everyone knows about them
    • probably in CONTRIBUTING.md
    • When the whole procedure is established, perhaps there could be an [ANN] post on discourse.
  • New contributors can ask the core developers to grant themselves a triage role
  • Core developers can also ask contributors who already meet the conditions if they would like to help with issue triage
3 Likes
9

An apocryphal story is programmers at the University of Michigan had to bring down MTS and they discovered a job that had been submitted 2 years earlier and never executed. This led to the development of aging schedulers in operating systems.

That’s one solution – the longer time has been since you’ve done it, the higher priority it should have.

Another solution is “spring cleaning” – once a year, the priority is to go through and clean up all the PRs.

1 Like
10

If Julia Computing can swing it, they should have someone where part of their job is to read all issues and connect the appropriate people based on priority. I try not to ping people that I know are busy, unless I think they will be interested or only they can address it. I’d rather core developers spend their time developing if possible and only answer issues when specifically contacted because they are needed.

11

Julia Computing - has No Governance Role
so the Julia Stewards - will be better.

EDIT:
Julia Computing - No Governence Role
“There are organizations that people sometimes mistakenly believe have some kind of official control over the Julia project. This section details two of them, but please note that neither has any official relationship to the Julia project, nor any governance capacity with respect to the project. Individuals who have some control over the project and happen to belong to these organizations, have that control as individuals and not as members of those organizations and takes the separation of those roles quite seriously.”

12

But the Julia stewards are developers and ensuring every issue is addressed is not typically a hobby for most people. My point is that this would actually get done if someone had a salary tied to it and it would be preferable if it wasn’t someone that would otherwise be doing work that only 10-20 people on the planet can do.

2 Likes
13

Pinging appropriate people is not Governance

1 Like
14

I think that a good labeling system with different tiers of necessary core developer involvement would be good. I think there should be a larger group of people allowed to approve changes to less critical aspects of Julia. Critical to me means correctness and API additions, because correctness is always paramount and API additions cannot be reverted later.

The lowest threshold category to me is docstrings and docs. It should be easy to add those and easy to get them approved.

The next one would be performance improvements that don’t change how algorithms work. If tests pass and performance is measurably better, that should be easy enough to merge.

After that would come algorithm changes that are harder to grasp in their effects on existing code. Then new small features, then new large features.

And the highest priority should be assigned to correctness issues, where damage needs to be contained quickly.

So to me capturing aspects like that in labels that any user can assign to their PR with a checklist would already make it much easier for the core people with the best overview to be freed from reviewing the less critical but still important improvements.

2 Likes
15

In terms of immediately actionable steps, it might be good if someone who has thought this through proposes a well thought out issue and PR pipeline/labelling system based on this sort of criteria. I don’t think we’re ever going to have a perfect solution because issues and PRs don’t always fit into convenient bins, but maybe a way to make the path forward more clear for people.

2 Likes