Meta-point in response this remark by @skoffer (with which I agree): with other languages, say Python and Go, more structured processes to discuss possible language evolution and improvements have been put in place over the years, e.g. Python’s PEPs. As probably is known by the Julia devs, each Python PEP lists a single proposed change, including rationale, discussion on pros and cons, implementation strategy, etc. This might be a model that can be of interest to Julia development at some point (perhaps still too early) to have more structured discussions and ways to get feedback on a clear proposal. It would also force those proposing changes to come up with a more concrete outline, which can be debated more sharply. A light-weight start could be to have a forum thread per proposal in a separate part of this discourse, which is moderated to make sure it stays on topic.
How would that be different from GitHub issues?
I guess it depends a little bit on how the github is currently used and its audience, compared to this Discourse, when it comes to discussions. To me issues on github (in general, not just with Julia) usually feel more developer-oriented and less user-oriented. A post here will also reach a wider audience I suspect, as not everyone closely follows activity on github. So I assume you will get different feedback and discussions depending on the platform chosen. Also, I personally find Discourse to be nicer for holding discussions, due to easier quoting and searching.
Python seems to have chosen for their Discourse forum to discuss PEPs (PEPs - Discussions on Python.org), by the way. But that might have something to do with them still using their own issue system instead of github.
I think perhaps you do not mean a PEP-like process for changes.
You mean some particular feature of the PEP process.
It is a big processes with many parts
It includes so many things.
From how things are discussed, to requirements around reference implementations, to processes of review, to the 13 sections required in the document, etc etc etc.
I doubt you want that whole thing, it is a huge huge process.
I suggest reading that full 14 page document, coming back here,
summarizing the whole thing, and highlighting with in that what things you think are good.
Otherwise, we will end up talking past each other, and being unclear what each is actually talking about.
(and not to be too persnickety, but if that sounds like too much work, consider that that is much less work than you might be suggesting imposing on the proposer of a new language feature)
I would also point out that a lack of input from different people is hardly the problem with the issue on which this was proposed (implicitly loaded modules). I don’t see how more heavyweight process would help in this particular case.
I’m not saying copy Python’s PEP process, but what’s wrong with having a process to suggest, discuss and decide on changes that currently can lead to lots of unstructured discussion?
The process would force more structure. Instead of having an open discussion about “something” it would force the creation of a (draft) proposal first to some level of concreteness which can then can be discussed and decided upon.
Also, I noted in the original post that it might be too early in Julia’s lifetime to move toward such as process. If the current ways in which these kinds of things are discussed and decided are still appropriate then there’s apparently no need (yet).
How would the middle “discuss” step of this process differ from “lots of unstructured discussion”?
As @Skoffer put it:
Sorry for interfere, but this thread is so long and there was so many ideas, that is really hard to understand what features are proposed.
Having a focused concrete proposal as a start would probably save from many of the side-discussions currently happening.
My reading of “Suggestion: use PEP-like process for changes”
Was that you were suggesting we used a process that was (very much) like PEP for changes.
I see where you are coming from with this.
As the counter to this, sometimes things just need a many month long unstructured discussion to settle on something that is good.
before a concrete proposal can be written.
And then at the time such a proposal can be written there is less point writing it, as you have all the answers.
Examples
- Taking vector transposes seriously
- Taking Complex Autodiff Seriously in ChainRules. GitHub1, GitHub2, Discourse
You might say: “These are exactly what we want to avoid”.
But they are not a waste of time:
They are not overhead blocking important work: they are the important work.
No one person could have come to the conclusions with the robustness that that long unstructured discussion has, because it is dealing with the intersection of multiple different fields requirements and expertise. (Its a cool thing that the julia community can bring to bear on a problem: combined centuries of expertise from dozens of fields)
We have to be careful to balance the notion of anyone being able to contribute ideas and understand what is being talked about, and having a clear focused proposal,
against experts being able to discuss vigorously between themselves, ranging across wide areas, to determine what should be proposed.
I’ll chime in here to note that there’s at least two communities a design process can serve and different processes are felt very differently by those communities:
- People who can participate in the full conversation and will keep up with each individual step that leads to a change in the overall consensus position.
- People who can’t participate in the full conversation and just want to know what the current consensus position is.
Unstructured discussions absolutely are needed, but they’re too hard for community (2) to keep up with. Therefore, if you only have unstructured discussions, you minimize the load on (1) but weaken (2)'s ability to understand where things are headed. In contrast, if you focus on continually updating a single artifact that reflects the consensus view, you add a lot of work for community (1) that provides them with literally zero benefits, but you can keep (2) better informed.
Even something light weight like always updating a verbal summary in the first post in a GitHub issue can go a long way. Indeed, this is so natural people often do it without it ever being a formal requirement. But whether such a summary exists usually is the difference between whether (2) has any idea where (1) is headed or not.
I’d also note that there is a framework that was seemingly created to host discussions on “Julia Enhancement Proposals”:
However, it seems that it was not the right solution, because there is nobody using it anymore.
It should perhaps be pointed out that a more heavyweight process like the PEP process, rather than “reach[ing] a wider audience” and getting input from more people tends to raise the bar for participation in the design process, rather than lowering it. If we had a more formalized process, only people who took the time to learn and follow the process would have the chance to participate.
I think the unstructured discussion is nice at first because it isn’t always clear what ideas are good or bad or need lots of discussion or don’t… but that perhaps at a certain point it should move beyond that. I don’t know that a formal process is needed though because the next step is really “make a PR” and you see tons of posts saying just that (or “make a draft implementation in a package”). The impetus for the OP was likely one of the 2 current megathreads, maybe the problem arising there is that the changes are breaking and so a draft PR won’t cut it. Maybe the Milestone tab on GitHub is the right place to store these things so they get revisited?
I think this thread should be closed (and perhaps never should have been split off). I would much rather see the Julia leadership make a pronouncement that the above PEP-like idea is rejected, together with a clear rationale, instead of arguing it all the way without coming to a conclusion. This thread is now very slowly turning into something that is just as exhausting a discussion as the original thread I mentioned the idea in. And the more time I spent on this forum the more I find discussions go in that direction and that just isn’t worth the drain on my energy and positivity anymore.
My suggestion is for you to take care of yourself and stop following such discussions if this is how you feel about them.
I like such discussions and would like see them to continue.
EDIT: Of course, being the OP you can request the Stewards to close the thread, and if other people feel like they want the discussion to continue, they can open a new thread, it is just a shame to have the history broken into two threads.
That may not be a bad thing. Input from a wide range of stakeholders in an issue is important, but so is a low signal / noise ratio. Some issues on Github keep meandering and get 200+ replies which a conscientious reader has to sift through or risk missing important points.
I am not saying PEP represents some ideal in this respect, but keeping some discussions a bit more focused could be helpful.
Also, Julia is fairly mature at this point, which itself implicitly raises the bar for improvements — it just ain’t that easy to improve any more without quite a bit of investment.
The reason for PEPs seem to be a language deficiency IMO. For Lisp-like languages you can easily extend the language from the language. The way a lot of this would be implemented is just standard Julia code in Base, or many times it would be a module/function/macro in Base that would just be loaded by default. So the best way to propose such a language change is to just make the package and show people how great it is by showing how many people use it. “Is this hard to maintain?” “Will people use it?” Easy answers: yes, and people do.
There’s some nice examples of this already having been done. For example, Parameters.jl was created and after years of adoption its general idea, @with_kw struct, was added to Base. Some of the abstract interpreter ideas started out as pieces like Cassette and IRTools before finally getting a more canonical implementation in Base (at a different IR level, because of the issues found when actually trying to use it). Versions of threading, Tapir, etc. MLIR is being demonstrated in Brutus.jl before adding it to the compiler.
Even the feature in question has an implementation in a repository, https://github.com/Roger-luo/FromFile.jl . It has seen essentially no adoption (JuliaHub statistics). The question shouldn’t be “who do I talk to in order to get a sign off in order to shove this into Base”, it should be “what needs to be changed so that it gets community adoption and everyone wants it in Base?”. If you look at the JuliaHub statistics for Parameters.jl, you see 238 direct dependencies even after the functionality has already moved to Base (so it used to be higher!), meaning that it demonstrated something that people wanted and it became no question that it should be added to Base.
If you look at how that turned out:
- Pkg3 was demoed in a separate repo before the new Pkg.jl was made as the default way to do package management in Julia
- Chris Foster took https://github.com/JuliaLang/Juleps/blob/master/Logging.md and made GitHub - c42f/MicroLogging.jl: A simple logging API for julia as the prototype, which then moved to Base after being successfully demonstrated.
- Nullables had a big successful discussion, https://github.com/JuliaLang/Juleps/pull/21, which then led to the missing change after trying ideas like DataValues.jl and Missings.jl in packages (and then optimizing the compiler for it after it moved).
- Structured concurrency (Structured concurrency by c42f · Pull Request #56 · JuliaLang/Juleps · GitHub) had a Tapir implementation (Experimental Tapir support by vchuravy · Pull Request #31086 · JuliaLang/julia · GitHub) which was the prototype that led to Takafumi’s changes.
Note that the last one is interesting because it’s actual compiler level change: most features aren’t. Most things can be done with a macro. In Julia, most things can be demoed as a package.
In open source, code is currency. If you think the feature is great, make it and have people adopt it. Then say “this should be the default in Julia”. That should be the last step, not the first.
Imagine how I feel 
My point of view is that our scarcest resource is time and energy of core Julia contributors. And there are a lot of things that need to be done. I personally have a seemingly endless list of things that I’d like to work on should I ever find the time. Compare that with an old and relatively slow-moving language like Python. I get why Python devs feel like they can afford a slow process with a lot of overhead: there is no urgent need for new features and most of the bugs and rough corners have been smoothed by decades of use. The language is also incredibly well established. They can take as long as the want to to develop new features. There’s no urgency and no lack of people to implement them.
I do not feel like we can afford that. Maybe in some years but not today. At least not if some Julia dev who already doesn’t have enough time is expected to manage that process. That said, if someone else wants to manage a PEP-like process, then that would be ok with me. Does anyone want to do that? @paulmelis, would you be willing to be the Julep process manager?