Policies around registering AI-created packages in General?

[Mikhail_Kagalenko]

@mbauman

The beautiful thing about a discourse board, @Mikhail_Kagalenko, is that you needn’t ask permission! I’ve split this more general discussion around General’s policies to its own thread, leaving some of Tachikoma’s specifics in its announcement thread, but in the future you can also just create new topics yourself

Makes sense, and also tells me the Discourse software has been designed by a
thoughtful human, rather than vibe-coded!

[Ronis_BR]

@Mikhail_Kagalenko

I would like to draw your attention to the fact that growing
number of OS projects ban altogether LLM-generated
code. My proposal of disclosure is far less radical than that.

I agree to add a guideline to show if the package uses vibe code, but I won’t make it mandatory. After all, what’s the real point? How can we really tell if Claude is better or worse at programming than developer X? I’ve even met folks who are much less skilled in programming than Claude! I am pretty sure we can find very bad packages in Registry right now.

At the end, all packages in Julia general Registry comes with no warranty. Vibe coded or not, you need to check and test if the package produces what you want.

@Mikhail_Kagalenko

Makes sense, and also tells me the Discourse software has been designed by a
thoughtful human, rather than vibe-coded!

Vibe coding something and planning what to code are two different things IMHO…

[Mikhail_Kagalenko]

@Ronis_BR

That’s debatable! If I learned C++ by diving into GPL tutorials and then used that knowledge to build a commercial product, would I need to share the source code?

No, but human learning and LLM “learning” are different in kind.

[langestefan]

@Mikhail_Kagalenko

I would like to draw your attention to the fact that growing
number of OS projects ban altogether LLM-generated
code. My proposal of disclosure is far less radical than that.

I’m not seeing that at all. In fact, the number of repositories that make use of claude has exploded. Most of the comments you made in this thread seem to be based on feelings, not facts…

[Mikhail_Kagalenko]

@langestefan

I’m not seeing that at all. In fact, the number of repositories that make use of claude has exploded.

That in no wise contradicts my point. I have not stated that the projects banning
the “AI” tools forms a growing share of all projects.
The amount of spam and slop will, without a doubt, always exceed the quantity of valuable
stuff.

[Ronis_BR]

@Mikhail_Kagalenko

No, but human learning and LLM “learning” are different in kind.

Why? At the end, we get information, make correlations, and adapt our current knowledge to a new problem. That’s not exactly how LLM training/execution works, but it is also not that different.

@langestefan

I’m not seeing that at all. In fact, the number of repositories that make use of claude has exploded. Most of the comments you made in this thread seem to be based on feelings, not facts…

Yes… In fact, in the Julia repository we have right now, Claude is one of the top committers. Were the PRs vibe coded? Was it only English check? Nobody knows… What about other LLMs that do not add the Co-Author information in commits?

@Mikhail_Kagalenko

That in no wise contradicts my point. I have not stated that the projects banning
the “AI” tools forms a growing share of all projects.
The amount of spam and slop will, without a doubt, always exceed the quantity of valuable
stuff.

I see this in very big, sensible projects like OSes. I am not aware of a community like Julia that has banned or put guidelines in what people must or must not use while coding their own packages.

[langestefan]

@Mikhail_Kagalenko

The amount of spam and slop will, without a doubt, always exceed the quantity of valuable
stuff.

What is slop exactly?

[Ronis_BR]

@Mikhail_Kagalenko

The amount of spam and slop will, without a doubt, always exceed the quantity of valuable
stuff.

Yes, sure, but it is up to the maintainers of the project to decide if they accept it or not.

We are facing the other, much more difficult side: lack of contributors!

One very important example: I created PrettyTables.jl, and it has been mostly a one-man show. However, last month, an amazing developer used AI tools and created a Typst back-end people asked me for years. It is working perfectly. Now, Julia has a good way to write Typst tables that would not have been available if LLMs were not a thing. In this case, do I need to add a badge saying that PrettyTables.jl is “vibe-coded”?

[mbauman]

@Mikhail_Kagalenko, I’d encourage you to pay particularly close attention to @Goerz’ comments as he is a core maintainer of the General registry. In particular, see that there is a policy about vibe-coding:

@goerz

We have a policy against vibe-coded packages, but it’s neither realistic, useful, or enforceable to ask that “no LLM has ever touched any part of this code”. I would expect there not to be any actively developed packages in the near future that would qualify for this. People are of course encouraged to disclose in some form of their choosing when they use LLMs as a very significant part of their workflow. But at the end of the day, our expectation is that humans have ownership their code. You can use an LLM, or any other automated tools such as linters and formatters, to write code or tests or documentation on your behalf, but you’re responsible for whatever you commit.

The guidance on General links to a prior discussion here:

Should General have a guideline or rule preventing registration of vibe-coded packages?

Note: here vibe-coding means “building software with an LLM without reviewing the code it writes”. Background: General is moderately but not totally permissive Package registries across programming language communities come in various flavors of permissiveness. Some, like npm/pypi/cargo, automatically & immediately register a package once a few automated checks pass. Others, like R’s CRAN, or TeX’s CTAN, perform a manual review which can take a few days up to weeks. As an example of the stricte…

[Ronis_BR]

@mbauman @goerz

@goerz

ou can use an LLM, or any other automated tools such as linters and formatters, to write code or tests or documentation on your behalf, but you’re responsible for whatever you commit.

What responsibility? AFAIK, all open source licenses state clearly that the software comes without any warranties or liability for damages. The responsibility lies with whoever is using the open source, free code.

[goerz]

@Ronis_BR

What responsibility?

Your moral responsibility to the Julia community. This is not about legal terms, this is about our expectation of packages in the General registry passing a bar of being maintained in a sensible way. We don’t want to be buried in packages that look good on the surface but break as soon as you poke them even a little bit (aka “slop”), nor do we want packages that are unmaintainable because no human developer has a clear picture of their design (aka “vibe-coded”)

[Ronis_BR]

@goerz

Your moral responsibility to the Julia community. This is not about legal terms, this is about our expectation of packages in the General registry passing a bar of being maintained in a sensible ways. We don’t want to be buried in packages that look good on the surface but break as soon as you poke them even a little bit (aka “slop”), nor do we want packages that are unmaintainable because no human developer has a clear picture of their design (aka “vibe-coded”)

Hum, not good. There are a lot of packages in the Registry that were not vibe coded, that are unmaintained, that do not work, that have breaking changes in minor releases, and probably would be better if AI was used so that the maintainer can do things easily…

Registry is not curated. Besides very few checks, anyone can submit packages. So we need to change this or let each developer decide what they want.

[kahliburke]

@goerz

We don’t want to be buried in packages that look good on the surface but break as soon as you poke them even a little bit (aka “slop”), nor do we want packages that are unmaintainable because no human developer has a clear picture of their design (aka “vibe-coded”)

How many of the >12k packages in the general registry currently compile on the latest and LTS versions of Julia, have tests that pass, over 50% unit test coverage, are well documented, and have been maintained (perhaps the bar being, have addressed any identified GitHub issue within the past 6 months)?

Have you tried Tachikoma.jl?

[mbauman]

Yes, the General registry is not curated. But there’s a key exchange that happens:

• By registering a package, you’re taking something valuable from the community — a name in the default Pkg registry.
• In exchange, you’re providing functionality!

Many folks here take the registered packages seriously — core devs evaluate test errors on new releases, many other package maintainers help ensure version upgrades work smoothly throughout the entire ecosystem, I’m currently working on tooling to help automate security advisories and updates, and so on.

Even in the most permissive of ecosystems, it’s still a major community problem when a package has serious issues.

[goerz]

@Ronis_BR

Registry is not curated. Besides very few checks, anyone can submit packages

That is not entirely true. There are both automatic and manual checks for packages that are submitted for registration, see, e.g., the discussion in

When does it make sense to post a new package to the Julia repository?

To be registered, it should be at a “minimum viable package” stage (which is what a v0.1 release typically indicates). This means it should have some non-trivial functionality. It also should have some tests, and very importantly, some form of documentation. This doesn’t mean that a package needs a full Documenter-based website: for simple or early-stage packages, putting the documentation in the README if usually sufficient. It should include a description of what the purpose of the package is,…

@kahliburke

How many of the >12k packages in the general registry currently compile on the latest and LTS versions of Julia

Hopefully many, as Julia Computing runs automated checks about this whenever releasing new versions.

@kahliburke

have tests that pass, over 50% unit test coverage, are well documented

There are minimum requirements for packages at registration time, and stricter requirements for 1.0 versions. I would be very open to making automated checks for new registrations stricter than they currently are. Running tests and checking for 50% unit test coverage in particular is something that’s been long on my mind. That’s a discussion to be had separately, though

@kahliburke

have been maintained (perhaps the bar being, have addressed any identified GitHub issue within the past 6 months)?

There is an expectation for registered packages that they continue to be maintained. If they become unmaintained: if nobody is submitting PRs and they continue to work, not problem. When PRs are submitted but not reviewed or merged, as registry maintainers we actively facilitate new co-maintainers being added to old projects, in whatever form is appropriate. In the most extreme version of this, we have a form of “eminent domain” where an unmaintained package is forked to a new organization with new maintainers, and the the registry is modified to point to that new fork – even without the involvement of the original package author (if they were hit by the proverbial bus and have been unresponsive despite several months of communication attempts across different channels)

The bottom line is that the General registry is not free-for-all (as discussed in the intro to the previous discussion about vibe-coded packages), but is a shared community resource

[kahliburke]

Cool, sounds like you’ve got it all covered then. Anything I’ve submitted to the registry is high quality and being actively maintained.

Although I’m still curious what actual percentage of the 12k packages pass the bar I proposed.

[Ronis_BR]

@mbauman

Yes, sure! Me too! Those are very important things that probably most of the devs look for. I just do not understand this “you shall not send AI vibe-coded package here”. You can have both! Actually, IMHO, it is easier today to meet those requirements if you use AI.

@goerz

That is not entirely true. There are both automatic and manual checks for packages that are submitted for registration, see, e.g., the discussion in

By “curate” I mean that someone is downloading the package, running some code, checking if documentation is OK, etc. This does not happen to a very high percentage of the registered packages. Let alone the new releases. Nobody checks for breaking changes in API and other things.

However, that’s fine! We are developers devoting our free time. But since we are developers devoting our free time, Julia community should be careful with those kinds of rules.

[mbauman]

@kahliburke

Although I’m still curious what actual percentage of the 12k packages pass the bar I proposed.

For passing tests (against julia-dev), it’s about 50%:

[Ronis_BR]

@mbauman

For passing tests (against julia-dev), it’s about 50%:

Did this plot remove packages with tests like:

@test true == true

?

[goerz]

@kahliburke

Have you tried Tachikoma.jl?

Just to be extra clear, again: I’m not seeing any issues with the Tachikoma.jl package from which this discussion spun off. You definitely seem to have sufficient experience to use the agent-workflow effectively. So: more power to you!

There’s absolutely no issue with LLM-assisted coding. It’s great when it can make people more productive, and I also agree that when used correctly, it has great potential to improve the quality of packages. For Tachikoma in particular, that seems to be the case.

What we’re trying to protect against at the ecosystem level is huge quantities of low-effort code swamping the registry.