General registry T&Cs and a warning to noobs (like me)

First, a disclaimer…

I :heart: Julia lang. Personally, I am heavily invested in Julia and have been for the past 5 years. As an entrepreneur, that is not a small thing. I am also a Julia evangelist. I promote Julia any chance I get. I’m happy to do so and will continue to do so.

As much as I :heart: Julia, it doesn’t mean I am immune to unpleasant experiences, so I am sharing these thoughts with good intentions hoping to impact some positive change.

Noob

Despite my age and despite how long I have been a part of this community, in many respects, I still consider myself a “noob” when it comes to open source software.

For me, February was a rare and beautiful month. I was able to spend pretty much the entire month working on Julia open-source projects. That also gave me an opportunity to move up the learning curve a bit when it comes to open source generally.

Registering your first package

If you are like me, I was excited to register my first Julia package. I keep joking that we should gamify your Julia journey in which case registering your first package would definitely be “levelling up”. However, if you are a scientist, engineer, or financial “quant” (or anyone for that matter), it is a good idea to pause and think about what that means.

Registering a package is a one-way street

My unpeasant experience came about recently as I am trying to build up the JuliaFinance organization.There is one small package that is core to all others. Every JuliaFinance package will likely have a dependency on this package. For various reasons, I decided to create and maintain my own registry for JuliaFinance. The amazing new package management system makes this possible and when I read the docs the first time, I immediately had the idea to maintain my own registry some day so I took the plunge last month.

Since all my new packages are (and will likely be) registered on the JuliaFinance registry, but this one core package was registered on the General registry, it made since to me to move it from General registry to my own registry.

In hindsight, I can appreciate that most of the ensuing unpleasantness was due to my own ignorance about how open source and registries, in particular, are supposed to work.

Although I am pretty sure not a single person on this planet is using the package I wanted to remove from the General registry (and if there was, I’d be happy to personally work with them to fix things), the push back I got was shocking to me. I can understand the conservative position of the General registry maintainers, but this was not a highly popular package by any means. It seemed like a non-brainer to carry out my request.

So here is my warning to noobs (like me):

Before registering a package, make sure you understand the ramifications.

General registry Terms and Conditions

I’m sure many people remember this story:

The story actually rhymes somewhat with my experience. It started with wanting to rename a package. Then, during the unpleasant episode, at one point, I admit I considered a nuclear option of hiding the git-sha-tree so the General registry would be broken for my package and Pkg would find it on my registry instead. I don’t consider myself a rebel or a community destroyer by any means, but the fact that the thought even crossed my mind means there is a risk. What would someone else do in a similar situation?

The episode concluded when I agreed to park the discussion until the transition from METADATA is complete, but the subject will likely come up again since I am still interested in unpublishing it.

When I read npm’s “Unpublish Policy”, it helped me to understand the position of the General registry maintainers. Like I said, the unpleasantness stemmed mostly from my ignorance of how things are supposed to work.

Then again, why didn’t I understand how things are supposed to work?

I never agreed to a Terms & Conditions when I registered my package. There are no Terms of Use for General registry.

As we prepare to migrate from METADATA to General registry, I think a part of that process probably should be the introduction of a T&C explaining, among other things, the unpublish policy.

Personally, I would be very happy if, upon the introduction of the T&C, we were given the opportunity to “opt out” and unpublish - without penalty - an existing package if unpublishing would not have a major impact to the ecosystem in ways that can be quantified, e.g. number of other packages that depend on it, etc.

That way, if I choose not to unpublish or if I publish a new package in the future, the terms of doing so are clear. They were not clear to me and I suspect they are not clear to many other package developers (who may not be OSS experts) as well.

Note: I was warned that I could be non-whitelisted for registering any new packages in the future if I insisted to unpublish, but I feel that would be unnecessarily harsh given there were no T&Cs - at least none that I was aware of - in first place.

Thought on an Unpublish Policy

npm’s unpublish policy is localized in time, i.e. you have a 72-hour window form the time you publish a package to unpublish. I hope the maintainers of General regstry might also consider localization in space, i.e. if a package is not a dependency for a large number of other packages, it can be unpublished. All this stuff could be clarified to help resolve the inevitable future disputes that will arise.

In any case, I am confident all of this stuff will be worked out in time :pray:

1 Like

It would be great if you could clarify about this, eg with a link to the relevant discussions. Readers unfamiliar with your story (eg me) have a hard time understanding what happened, and the details are very important in this case.

Also, because of the lack of details, your post does not really help with

Hi @Tamas_Papp,

details are very important in this case.

There is not really a case here. My point is twofold and I don’t think either point requires rehashing some umpleasant details:

  1. People should be aware that they are giving up some rights when they register a package.
  2. We should have a proper T&C to spell out exactly what rights we have and don’t have.

Now that I better understand the “norms”, I don’t think there was anything wrong with the way my situation was dealt with. Let’s look forward :pray:

Edit: The right I didn’t understand I was giving up when I registered was the right to “unpublish”, but now I do understand that.

1 Like

But there is — I thought you were describing a specific sequence of events involving registration and then de-registration of a particular package.

If the discussion happened in public, providing a link to it would help to make your post more concrete. Without details, it is rather vague.

I found the message “there is no de-registration process” quite clear and concrete.

It is not known how many peoples code rely on a published package, regardless of the number of dependencies it has. No package has OhMyREPL.jl as a dependency but I know it has quite a few users. The reason an open source license is enforced for registering in General is exactly so that we can (almost) give a guarantee that the package will continue to be available. That’s the beauty of open source that you opted into by choosing (or taking over maintenance) of a package with an open source license.

Depending on the license you chose, someone can literally copy paste your package and register it. So it is not about registering it, it is about the license you put on your code. So I would tweak this to:

Before choosing a license for your package, make sure you understand the ramifications.

5 Likes

Can you please provide a link to the source of this message? I am genuinely confused, there must be a discussion or document I have missed completely.

It seems to me that the OP’s intent is not actually to unregister their package, in the sense of removing access to it. Rather, that package should be moved from one registry to another.

I guess it’s possible to have the same package (or: two packages with the same name) registered multiple times (in different registries). There is then a potential (and unavoidable?) nameclash when users subscribe to multiple registries.

Then again, for the use case described by OP, this confusion could be resolved by keeping the old package in the General registry and version-bumping the relevant package in the JuliaFinance, together with all dependend packages.

Also, I should think that the UUIDs that are used in Project.toml should help with these kinds of clashes?!

EDIT: My point is: there seem to be technical solutions to this communication problem of unintentended consequences.

3 Likes

Thanks @kristoffer.carlsson,

It was with your help, I managed to fix the issues I was having with the package in question.

This case is a little different, I think, but you raise a good point that will help educate people (which is the purpose of this post anyway).

If I put an MIT license on my package, I understand I am giving people permission to use my code in their packages. I didn’t realize I was giving people permission to copy / paste my package and register it themselves against my wishes and I suspect I’m not alone. That feels slimy, but good to know. Like you suggest, we all need to better understand the licenses.

PS: One thing that I think might actually be a corner-case bug…

At the end of the day, I was ok to have v0.X registered in General, but I didn’t want future versions showing up on General. In other words, if someone requested v0.X, it would be found on the General register and other registered packages can depend on V0.X, but if they requested v0.X+1, it would not be found on General register unless I submit a PR to METADATA / General even though there is a v0.X+1 tag on Github.

However, I noticed General was picking up the new tags even though I didn’t register the new tags with Attobot. This feels like a bug.

I think we should be able to tag versions and have the tags not show up in the General registry if we don’t want them to.

It seems to me that the OP’s intent is not actually to unregister their package, in the sense of removing access to it. Rather, that package should be moved from one registry to another.

Yes! Exactly :slight_smile:

I guess it’s possible to have the same package (or: two packages with the same name) registered multiple times (in different registries). There is then a potential (and unavoidable?) nameclash when users subscribe to multiple registries.

Yes. Pkg currently deals with this situation with a prompt asking the user which one they want. This is kind of cool, but a better solution would be to allow people to specify the registry when they add a package.

Then again, for the use case described by OP, this confusion could be resolved by keeping the old package in the General registry and version-bumping the relevant package in the JuliaFinance , together with all dependend packages.

Yes! Exactly :slight_smile:

I would be happy with this solution, but there seems to be a bug at the moment that prevents this from working properly.

Also, I should think that the UUID s that are used in Project.toml should help with these kinds of clashes?!

Yes. These clashes are already handled by Pkg :slight_smile:

The general registry does a straight up conversion from METADATA so that sounds odd. Are you sure that this is what happened?

That is totally fine from the point of view of General. Anyone adding the JuliaFinance registry would then get the updated versions since multiple registries are unioned.

2 Likes

On the other hand, a short page explaining such things and processes can’t hurt.

Apart from the deregistration process, it would be nice if there was an “abandonement process” (no longer want to maintain) and some explanation of responsibilities (Hypothetical example: DON’T ship backdoors. DON’T just vanish from the face of the earth; INSTEAD trigger abandonment process. DON’T give up control of your package to friendly strangers from github when you don’t want to continue maintaining it; INSTEAD trigger abandonment process.).

5 Likes

The general registry does a straight up conversion from METADATA so that sounds odd. Are you sure that this is what happened?

I can’t say I am 100% sure, but I’m almost certain I turned off Attobot before tagging and the new tag still showed up on METADATA anyway. Not 100% sure though. I’ll be super careful next time I tag the package to be sure. It might be the case that it takes some time after you turn off Attobot before it actually deactivates? :man_shrugging:

That is totally fine from the point of view of General. Anyone adding the JuliaFinance registry would then get the updated versions since multiple registries are unioned.

It sounds like we found an acceptable solution to my particular case then :+1::blush:

grep for “unpublish”?

Not clear where I should search for unpublish, the Julia source tree yields nothing, nor do the general registry, the METADATA.jl, and the Pkg.jl repos.

Is a link too much to ask for?

A lot of the discussion happened on Slack and it is long gone by now.

Sorry, i’m in too many parallel discussions right now (incl. this office); i meant: Read about “unpublish” in the first message of this thread.

I see, the joy of discussing things with long-term consequences on ephemeral platforms. :confused:

3 Likes