Policies around registering AI-created packages in General?

[kahliburke]

@Ronis_BR

@test true == true

Have to test the compiler too!

[Ronis_BR]

@goerz

Just to be extra clear, again: I’m not seeing any issues with the Tachikoma.jl package from which this discussion spun off. You definitely seem to have sufficient experience to use the agent-workflow effectively. So: more power to you!

There’s absolutely no issue with LLM-assisted coding. It’s great when it can make people more productive, and I also agree that when used correctly, it has great potential to improve the quality of packages. For Tachikoma in particular, that seems to be the case.

What we’re trying to protect against at the ecosystem level is huge quantities of low-effort code swamping the registry.

That’s awesome and I fully agree! So, perhaps reword the README in Registry? Because I think that:

• Packages that are “vibe-coded” (generated by an LLM without human review) are not suitable for registration. See the LLM policy.

is too vague. What is human review? Go through each line? Only checking if the outcome is correct?

[goerz]

@Ronis_BR

What is human review? Go through each line? Only checking if the outcome is correct?

When you use an LLM, you “own” the code. It will be judged by the same standards as if you had submitted it by hand. And, if it shows signs of “slop”, you won’t get much of a benefit-of-the-doubt.

So “human review” means whatever you deem necessary to fully stand behind your package, as an author

[mbauman]

@Ronis_BR

Because I think that:

• Packages that are “vibe-coded” (generated by an LLM without human review) are not suitable for registration. See the LLM policy.

is too vague. What is human review? Go through each line? Only checking if the outcome is correct?

Did you click those links and find their definitions? They’re very human definitions, not machine-readable algorithms.

And, no, no assertion on test quality is made by PkgEval or nanosoldier. The goal of it is to do due diligence for package authors that have done theirs.

[kahliburke]

@goerz

Just to be extra clear, again: I’m not seeing any issues with the Tachikoma.jl package from which this discussion spun off. You definitely seem to have sufficient experience to use the agent-workflow effectively. So: more power to you!

Oh well that’s a relief, considering the statements you made yesterday in which you singled me out, and urged me to curb my enthusiasm. Good to know that I’ll not need to expect any more such admonitions from someone who hasn’t even used my package.

[ANN] Tachikoma.jl — A terminal UI framework for Julia

It’s also very borderline with respect to the requirements for registered packages! There’s a very fine line here between AI-assisted coding and code deployed by LLMs without human intervention, aka vibe-coding, which is NOT acceptable for registered packages. While it’s great that you’re creating this framework, I would urge you to curb your AI enthusiasm just a little bit, and remember that the fundamental design, the quality, and reliability of your package are still your responsibility, an…

It sounds like you’ve got a few thousand other package maintainers who might need to receive a talking to from you, anyway …

[Ronis_BR]

@goerz

When you use an LLM, you “own” the code. It will be judged by the same standards as if you had submitted it by hand. And, if it shows signs of “slop”, you won’t get much of a benefit-of-the-doubt.

So “human review” means whatever you deem necessary to fully stand behind your package, as an author

Ok makes sense considering that “standing behind” is only moral and not legal.

@mbauman

Did you click those links and find their definitions? They’re very human definitions, not machine-readable algorithms.

Sorry, what links? I think I missed something here.

@mbauman

And, no, no assertion on test quality is made by PkgEval or nanosoldier. The goal of it is to do due diligence for package authors that have done theirs.

That’s the problem I have been talking about.

[langestefan]

@mbauman

By registering a package, you’re taking something valuable from the community — a name in the default Pkg registry.

Why is the Julia registry so different from pypi in this regard? Why do we care if a name is used?

[goerz]

@Ronis_BR

By “curate” I mean that someone is downloading the package, running some code, checking if documentation is OK, etc. This does not happen to a very high percentage of the registered packages

The automated checks handle downloading and installing the package.

I definitely glance at the documentation for every new submission, and a lack of documentation is the most common concern resulting in a (temporarily blocking) comment. Usually, the lack of documentation is easy to fix, and the registration can then proceed.

“Running some code”: Very occasionally I do run some of the example code in the README (in particular if I have a suspicion that it’s vibe-coded nonsense), but this is indeed rare. I would like to be less rare, and I think that it can be automated by requiring tests with a minimum (low) coverage.

I also wouldn’t mind if major breaking versions and 1.0 in particular had a waiting period with the potential for stricter automatic and manual review. That’s a thorny issue, though, because it might raise the bar for packages to go to 1.0, when we really want more packages to get off 0-based versioning (some people even think we should only allow 1.0 packages in the registry).

So these things will require more internal discussion on how/if they can be realized. My feeling is that is the Julia ecosystem grows, we are moving towards more curation and higher quality standards for registered packages (and also stricter naming guidelines), and that’s probably a good thing.

[goerz]

@langestefan

Why is the Julia registry so different from pypi in this regard?

Community consensus. Or, strictly speaking, registry-maintainer consensus. I’d again point to the older thread about Background: General is moderately but not totally permissive

[Ronis_BR]

@goerz

I also wouldn’t mind if major breaking versions and 1.0 in particular had a waiting period with the potential for stricter automatic and manual review. That’s a thorny issue, though, because it might raise the bar for packages to go to 1.0, when we really want more packages to get off 0-based versioning (some people even think we should only allow 1.0 packages in the registry).

I really think you are missing a point here. I have seen many minor updates with breaking changes. There is no way today to spot if a maintainer changed the API and submit a minor release.

Just so you know, there are much bigger issues facing our community than “this package is vibe-coded.”

[mbauman]

@langestefan

Why is the Julia registry so different from pypi in this regard? Why do we care if a name is used?

PyPI also cares about the packages in its registry, albeit to a lesser extent. It restricts registration of some names, allows for reclaiming abandoned names, and will even unilaterally quarantine seriously problematic packages.

[kahliburke]

@goerz

I also wouldn’t mind if major breaking versions and 1.0 in particular had a waiting period with the potential for stricter automatic and manual review. That’s a thorny issue, though, because it might raise the bar for packages to go to 1.0, when we really want more packages to get off 0-based versioning (some people even think we should only allow 1.0 packages in the registry).

I think that would be a poor change to make.

That being said, my initial request for inclusion of Tachikoma.jl into the general registry was for version 1.0.0. I chose to do that because I felt it was useful enough and of sufficiently high quality to warrant that version number, as a complete, functional, well tested, well documented, fully formed package with utility to the Julia community.

[kahliburke]

@Ronis_BR

Just so you know, there are much bigger issues facing our community than “this package is vibe-coded.”

But it makes such a convenient target …

[langestefan]

Has it ever been considered to adopt the same model? I would be in favour of more freedom in package registration. I think the current LLM wave is just getting started, and it may burn out people who have to ‘gatekeep’ the registry.

[Ronis_BR]

@[ANN] Tachikoma.jl — A terminal UI framework for Julia

While it’s great that you’re creating this framework, I would urge you to curb your AI enthusiasm just a little bit, and remember that the fundamental design, the quality, and reliability of your package are still your responsibility, and not something you can delegate to an LLM. Moving fast and breaking things is not something that leads to a healthy ecosystem

Sorry @goerz, but now that I read this again, I need to publicly say that this kind of wording is what leads to an unhealthy ecosystem.

I have been trying to develop a TUI for Julia, like Rust has Ratatui and Go has bubbletea. Check here TextUserInterfaces.jl how many years I have been trying and I am not even close. What @kahliburke did is amazing. Tachikoma.jl has a great design, is easy to use, is full of features, and is very well documented. All of this done by one dev with AI.

So @kahliburke, do not slow down! Continue your amazing work. I have already adopted Tachikoma.jl here at the Brazilian National Institute for Space Research (INPE) as the framework to build the TUI for our satellite simulator.

[goerz]

@Ronis_BR

I have seen many minor updates with breaking changes.

If you are talking about packages > 1.0: those are bugs, and you should report them. We can’t possibly “police” that. It is the responsibility of the package maintainer.

This does not apply to packages < 1.0: According to semantic versioning (and people really should read that document, since there are a lot of misconceptions around pre-1.0 and post-1.0 releases) there is no stable API prior to 1.0. Anything can change at any time – or rather, it is up to the developer’s discretion how to manage their API.

This is made slightly awkward by Pkg treating 0.x.0 versions as “breaking” when evaluating compat bounds, as a pragmatic choice, as well as by the policy of the General registry to treat 0.x.0 releases as “breaking” and to require release notes. I feel that this is reasonable as a policy: Registered package are intended to be used as dependencies, and that requires some form of API, even it if is not “stable” (post-1.0) yet. In this context, the main difference between registered package with pre-1.0 version numbers and post-1.0 version numbers is how they worry about backwards compatibility: pre-1.0, the concern should be to improve the package API without much concern for compatibility. Post-1.0, backwards-compability should be a major concern, and breaking releases should be rare. In my opinion, at least.

[kahliburke]

@goerz

If you are talking about packages > 1.0: those are bugs, and you should report them. We can’t possibly “police” that. It is the responsibility of the package maintainer.

But would you be willing to scold them a bit, publicly, tell them that their behavior is questionable and that they should change their development practices? That’s all I’m asking for.

[mbauman]

In my personal opinion, Julia — especially with v1.11 — makes it so easy to depend upon unregistered packages that I find GitHub (or GitLab or Google or whatever) to be an even better “free-for-all” registry than a real registry could ever be.

[goerz]

@Ronis_BR

Sorry @goerz, but now that I read this again, I need to publicly say that this kind of wording is what leads to an unhealthy ecosystem.

As far as I can tell, Tachikoma is great! But it might also not be the most typical example for a package that uses LLM to such a great degree. We can’t measure all developers, or “vibe-coded” packages by its apparent level of quality.

I also need to separate the general policies (packages are evaluated on their merits independently of how they are developed, and “slop” is not tolerated) from what might be my personal advice.

Merging an LLM-generated PR with only an LLM review is a bad idea. I’ll stand by that. I suspect that @kahliburke is overstating this a bit, and that an extensive human-guided test framework and manual engagement with the issue and PR makes this much less problematic in this particular project, compared to letting LLMs run wild in other purely vibe-coded packages. But at the end of the day, we come back to the idea that it’s the result that matters, not what tools you use to get there – and by all means: use the best tools to get the best results.

Apart from that, the “curb your enthusiasm” is just personal advice… take it or leave it, and @kahliburke in particular can feel free to ignore it. After all, I have a very limited view on their background, experience, and workflows.

If people were to ask me for my personal advice on whether it is a good idea to use massive agentic coding to rapidly build big projects, I would always caution to “slow down” a bit. These new tools are very seductive, but they can easily outpace anyones ability to design, test, and reflect. For people who don’t ask my personal advice: I apologize for giving it unsolicited, but I don’t think @kahliburke is going to let me slow him down with a comment like that

@kahliburke

But would you be willing to scold them a bit, publicly, tell them that their behavior is questionable and that they should change their development practices?

I tend to be pretty happy scolding people a bit . I would definitely call out violations of semantic versioning, especially for registered packages, but also more generally: [docs] No documented breaking changes in v7 · Issue #776 · actions/upload-artifact · GitHub

[technocrat]

[withdrawn due to inadvertent violation of generative content]