Can we have pre-release versions in the registry? (v2)

General Usage
,
21

Im not sure I would say this necessitates restricting features for more mature libraries in the ecosystem. But if it’s a big concern, perhaps patch releases could be blocked on 0.x.y? Because at that stage, it is effectively a beta for 1.0.0 anyways.

I’m more interested in beta releases for 2.0.0 and later, for which there isn’t a great alternative at the moment. I’ve seen a few instances where a 2.0.0 is released and then there are rapid fire bug fixes when users start trying it out. Perhaps a 2.0.0-beta.1 could mean a few bug fixes get to be entered before it hits production.

2 Likes
22

I would not rationalize the lack of this feature, I think it is simply not implemented yet.

Pre-release versions for testing make sense above a certain complexity. Julia is the certainly the most complex piece of software in the ecosystem, but I think that now we are in a situation where it would be useful for some packages to have beta releases, too.

Yes, this was certainly not needed at the stage when the Julia ecosystem was in its infancy. But it may be needed now. I don’t see any conceptual problem with it.

6 Likes
23

This is nothing unusual and in my experience means that they will copy and paste whatever instructions they need to perform. It doesn’t matter much whether the instructions are clear or cryptic, but it does make a difference if they need to copy one or multiple lines, if the instructions need to be repeated, or if the instructions can fail.

I won’t say it’s a perfect solution and there may be Pkg limitations also for this approach, but I think it may be helpful to provide some more details on what custom registry maintenance entails.

Preparations

Add LocalRegistry to your global environment. (It can be in a custom environment but then you need to tweak the register instructions somewhat. It should never be a dependency of your packages.)

Setting up a registry

This is a one-time operation involving the steps of creating a new and empty GitHub (or service of your choice) repository, followed by running the LocalRegistry.create_registry function:

using LocalRegistry
create_registry("NAME_OF_YOUR_REGISTRY", "URL_TO_YOUR_REGISTRY", push = true)

Activating a registry

This is done by the package manager by running

using Pkg; Pkg.Registry.add("URL_TO_YOUR_REGISTRY")

This is the one line instruction your testers need to copy and paste to opt in to your pre-releases. They don’t need to interact with LocalRegistry in any way and the available package versions are automatically merged by Pkg between your registry and General, so to the users this is identical to pre-releases having been registered in General.

Registering a package

A new package is added to your registry with the LocalRegistry.register function. There are some options for how to do this, but the one I prefer is

  • Start Julia with your package in the current directory.
  • Run using LocalRegistry; register(registry = "NAME_OF_YOUR_REGISTRY).

Note: This takes into account that the package already exists in the General registry.

Registering a new version

Adding a new version is done in the same way, but you no longer need to specify your registry name. It is assumed, as usual, that the new version number is in Project.toml.

  • Start Julia with your package in the current directory.
  • Run using LocalRegistry; register().

Note: This replaces the Registrator invocation you would use for the General registry. The changes are effectively immediately; there is no automerge to wait for.

Final comments

  • Since all of this is done with Julia functions, you can easily script the making of pre-releases.
  • If you want to experiment with this approach, I’m willing to assist.
4 Likes
24

Unless I’m missing something, one problem with LocalRegistry is that,

  • either you register packages with beta versions v1.0.0-beta.1 in your local registry, but then Pkg must be able to parse such versions correctly.
  • or you register prerelease versions of packages with release versions v1.0.0 in your local registry. But I think this is not ideal because then the local registry is not consistent with the General one (i.e. v1.0.0 means different things in the different registries).

Maybe one workaround, is to register versions v1.0.0, v1.0.1, …, v1.0.94 in the local registry, which are all pre-release versions. And then only register the last one when you are happy with it in General as the “release version”.

25

This would (currently?) prevent registration PR automerge.

26

Here I’m talking about Local Registries, not the General registry.

27

You can register beta etc versions in a registry and Pkg can parse them and, as far as I can tell, resolve them. There is no way to register Compat on pre-release version numbers though. The General registry refuses to automerge them, but that is a policy question (which makes sense with the Compat limitations).

You should under no circumstances register the same version number pointing to different commits (technically tree hashes) in a custom registry and in General.

If you need to do workarounds with regular version numbers, I would propose to use high numbers from the previous iteration, e.g. 1.100.x as pre-releases for 2.0.0.

1 Like
28

(post deleted by author)

29

Thanks for the helpful writeup. Does LocalRegistry.jl work with pyjuliapkg? This is how PySR gets its Julia dependencies. It’s not clear how I would specify additional registries in the juliapkg.json file.

Btw @GunnarFarneback does LocalRegistry.jl let you get pre-release version resolution like described this comment?

Both of these would be required for this to be viable.

But assuming one can do this (custom registries in pyjuliapkg and prerelease version resolution), I think the user side of things for PySR would be essentially solved.

30

I have no familiarity with pyjuliapkg. My guess is that it would work out of the box with additional registries if it uses the normal Julia Depot (i.e. .julia) and might need a feature request otherwise.

The only thing LocalRegistry does is to put information into registries. Version resolution is handled by Pkg.

31

see How can I add and manage dependencies from custom registries? · Issue #34 · JuliaPy/pyjuliapkg · GitHub which was prematurely closed.

Registries are (unfortunately) global to the whole julia installation, not local to an environment. So you’d need to get the user to execute juliapkg.executable() to get the julia path, and then get them to run something like

$(juliapkg.executable()) -e "using Pkg; Pkg.Registry.add(...)"

Not anymore: Add back support for full VersionNumbers to the resolver · Issue #3963 · JuliaLang/Pkg.jl · GitHub

32

Thanks. I had a look and it seems it is not compatible with custom registries.

Basically, pyjuliapkg performs the following:

  1. Installs Julia,
  2. Creates unified Project.toml based on python dependencies’ requested packages, then
  3. Instantiates environment.

The incompatibility comes from it running all of these in a single step, when you call juliapkg.resolve(). This is what gets called when you do import juliacall. So there is no opportunity to inject a registry, and depending on versions that are not in General would fail.

Thx. So I guess we would still need the prerelease resolver feature regardless of choice of registry!

33

Cool! I’ll link this discussion on that issue.