Different rules for project control: It would be made clear that control of the “curated” version always belongs to the curation committee. Sure it is typically exercised by the project maintainers, but the curation committee should always feel free to point to a fork if they disagree with the project management (and users of Curated/foo.jl would automatically stay on the fork).
This is a very significant misread of open source maintainership and our capabilities as a community. We do not have a group of maintainers in want of unpaid work to do, nor the funding to make it a paid opportunity. One reason to make basic package forks difficult to register on General is that it helps to preserve the one nonfungible resource we have as a community: maintainers.
Curated journals do not imply future work, this is a completely different set of promises that publishing a package implies. Of course maintainership can be dropped at any point but there is a tacit assumption that when you register a package you will do some amount of maintenance/development for some time.
More information available to users (in an open source way ideally without relying on JuliaHub) like tags, repo activity, responsiveness etc is a better way to “curate” the registry because it relies on real historical information on the package and its developers.
That being said, if there are good reasons to fork, for instance if there are somewhat significant changes, which are well justified as a separate package, then register it. Then you are responsible for maintenance and informing the community why you are different.
The level of “ownership” is a gradual by-convention thing, even where it doesn’t impact legal considerations.
The General registry overriding a package author’s wishes is a nuclear option reserved for very gross misconduct (or extended absence / death).
On the more curated side, archlinux or debian applying custom patches to the software they distribute is standard procedure.
On the less curated side, can you imagine the outcry if google or apple decided to actually do real curation, and ship a software update that replaces a third-party app for all its android/ios users by an in-house developed clone/fork? Developers would cry and sue about “theft”.
But ultimately, I think the best world for users is the mix you see in linux distros: Have a heavily-curated core, together with a lightly-curated and an uncurated periphery. Install from core by pacman -S, install from lightly-curated via your favorite AUR-helper, install from uncurated by whatever applies between curl ... | sh or getting some tarball and the usual ./configure / make dance (in the windows world: download some setup.exe somewhere for no curation, whatever nuget ).
Today in julia, we have the lightly-curated General, the uncurated sea of git repos, and the heavily-curated base and stdlib.
I’m arguing that we need more functionality to move from light to heavy curation. And I’m arguing that this requires not just community effort, but also technical effort in Pkg support: A dependency on FooBaz.jl can be satisfied by either Curated/FooBaz.jl or General/FooBaz.jl; and users can explicitly choose which registry to give priority, both as a global default, and also for specific packages.
stdlib stability rules are probably a little too heavy. So we would want something in-between.
Disagreements like the one you mentioned occasionally happen. The suckage typically leads to people shaking their heads about curator decisions, and voting with their feet, i.e. making the active decision to trust the package author over the committee for this specific package.
For example the not-so-hypothetical julia archlinux case: People know that the archlinux-curated julia flavor has problems, so they instead use the upstream julia. Luckily this is easily installed (archlinux includes juliaup in its curation).
I can’t say anything about how annoyed either party (julialang upstream vs archlinux) got at the other. There used to be a bunch of archlinux specific issues on the bugtrackers, and some julia package maintainers got slightly pissed (“dude, stop installing the archlinux crap and then opening issues on upstream. Reproduce with stock julia or STFU. Issue closed.”; good advice, harshly delivered).
For me, as a user, the outcome is still good: “trust the committee over individual developers” scales very well as a default. There are rare exceptions (no institution is perfect), and in these cases I am empowered to override the curation for my system.
I’d like to point at linux distros.
In the vast majority of cases, linux distros don’t really modify the upstream packages they redistribute (and in the cases where they do significant modifications, the ultimate picture is very mixed). They act more like a distributed oversight.
But the trust relationship is different and intermediated. As such, the linux distro adds a giant ton of value for end-users through their set of curated packages.
Maybe, but we really don’t have the resources for “heavy curation”
More curation is more likely going to be in the form of stricter automated checks, evolving at a slow pace.
We already have that: you set up which registries your Pkg consults. The “General” registry is just the default that ships with all Julia installations. You can remove it, and you can add your own registries.
Absolutely not. This is never going to happen. It’s a core part of the registry design that different registries have to agree about information they provide about the same package UUID. The question of “priority” cannot come up, by design.
It’s a very nuclear option that has never been exercised as far as I know, and isn’t likely to be exercised anytime soon. This little community spat around the Kaimon.jl package is not something that comes anywhere near that threshold. We don’t do heavy-handed management of the General registry like that.
The point stands even there. Michael and a few others do a lot of work on what is fairly light curation. We have fewer people paying much attention to General registration than most major distros. If you’d like to make a curated registry nothing is stopping you, but if you want a community wide curated registry you have a lot of groundwork finding dedicated volunteers who are interested in that mission. And I tend to think fragmentation like that for the current issues is not worth it.
No, it has never worked that way. However, the automatic installation of General (*) only happens if you don’t already have some registry installed.
(*) To be more precise the default registries. Usually this is only General but if you are connected to a package server providing additional registries, all of those are automatically installed if you don’t have any registry installed. This is a bit problematic though if you happen to get only General installed before connecting to the custom package server. Then you need to explicitly do a pkg> registry add (without further arguments) to force installation of the default registries.
In one hand I’d love a curated registry, but on the other hand I fear it would lead to crystallization and basically relegate any package still in General to low tier scrap, stifling adoption and preventing discoverability just because a previous package got to the curated registry before the others and became the de facto default tool (although maybe not the best). I am trying to draw a parallelism with python if it wasn’t clear.
The network effect is a real thing and not always good in terms of innovation and striving for the global optimum at the expense for a “easier” local one. We’re social animals first and rational second.
Could there be something like a self registered package? A package which contains metadata about available versions, their dependencies and their git tag in its own repo? Then such a package which is specified by a git link could still participate in the normal version resolution including as a nested dep.
This would not guarantee reproducibility but might be valid way of herding small connected set of peseudo-private packages (like in a research group) while still being able to share Project.toml, use normal CI setups and so on without the complication of private/local registry on the enduser. Thus less convenience pressure to put stuff in General prematurely.
The metadata about available versions is definitely something I didn’t think of when I imagined resolution of unregistered packages; maybe that could be generated from a git repo, but if not, I think a local registry can do that job. The problem I mentioned earlier still stands: whether unregistered or in different registries, different packages can share a name (not UUID, very probably). Adding by name can become annoyingly ambiguous (“would you like Plots from General or CuratedGeneral or this random repo we found”), and projects can’t directly depend on 2 packages with the same name. That’s something a unified registry solves with unique package names. In hindsight, that might be why the package managers that load metadata on demand still primarily work on a single index/registry/channel at a time and resolve name conflicts with a user-specified priority of indices in the rare use cases.
I suggested embracing adding a few registries earlier, but it’s worth mentioning that registries don’t have explicit or automatic dependencies on each other and aren’t simply specified in Project.toml, so we’d just have to know “by the way, this package needs the General registry and Blah registry.”
Maybe if a CuratedGeneral is too much work (Generals’ reproducibility for lightweight environments is already exceptional), maybe a lower-effort and less reproducible registry for the less committed would address my initial question just as well. That could also help filter more committed open-source developers toward General. On the other hand, a EvenMoreGeneral could get so much larger that its minimal work, e.g. screening malware, outpaces General’s.
Have a look at LocalRegistry.jl . I use this to maintain a “PackageNursery” registry just for to make pre-General releases available for easy install via Pkg. It also could be used to maintain a sub-community package ecosystem.
However for users this comes with the need to trust me that I don’t register e.g. LinearSolve.j v6 in my own registry which would override the lower versioned official package.