The current METADATA release process

Announcements
1

As some of you might have noticed, the process of releasing and registering packages in METADATA has changed over last months. We have experimented with making the process more automatic such that new versions can be released faster and with less labor involved in the process. I’ve given some advice to the new process in individual PRs but there hasn’t been any announcements of what has changed. This is the purpose of this post.

The way things work now is that new releases are merged automatically every half hour provided that

  • The release is made with Attobot
  • METADATA Travis tests pass
  • CIBot reverse dependency tests pass
  • No comments have been posted after the latest Attobot comment
  • No labels have been attached

For new package registrations, the same rules apply but furthermore they have to stay open for three days. As a consequence of the automated process, you are now even more strongly encouraged to use Attobot for releases. Otherwise, the release process will typically be slower since the release will require actions from a METADATA maintainer.

As part of the transition to the more automated process, it is now expected that the package maintainers are active in identifying issues with their PRs by reading the logs. Once the issue has been identified, the author should update the PR, see GitHub - attobot/attobot: Julia package release bot, (and not create a new tag). Hence in general, METADATA maintainers won’t comment on (or merge) PRs with failing tests. If you are unable to identify why tests are failing then please leave a comment in the issue and we’ll try to help you.

Sometimes the CIBot tests takes time to finish. There can be several reasons for this. Most often it is simply because many packages have been released within a short period so there is a queue. For packages with many reverse dependencies, there will be a lot of packages to tests. Sometimes, the tests won’t finish because one of the packages hangs (we are working on adding a time limit). Finally, it sometimes, but rarely, happens that the process that handles the webhooks fails. If that happens, it requires actions from one of the CIBot maintainers. If you think the CIBot tests are taking too long then please leave a comment but remember that your comment will disable the automatic merging mechanism so please wait at least six hours.

Some packages can’t be tested by CIBot, e.g. if it depends on proprietary solvers of if the package requires special setup of the CI system. Such packages can be marked as untestable and the CIBot tests will be skipped. Please leave a comment in the PR if the package can’t be tested by CIBot. When possible, it is best to organize your tests such that CIBot is able to test your package since the reverse dependency testing often detects breaking changes.

Some packages require special setup when run through CI. In particular, some packages with Python dependencies need certain Python packages to be installed. CIBot uses Conda when installing PyCall so it is possible to install Python packages via Conda conditionally on the environment variable CI=true. As an example of how this can be done see https://github.com/tkf/IPython.jl/blob/6b4b7995b8d02dbd68f1a7fca95996a440fc988c/test/runtests.jl#L3-L5.

Finally, CIBot no longer has any issues with project files. There was a period when CIBot couldn’t handle project files and I therefore asked people to delete project files from packages. This is no longer the case. However, the General registry is still based on METADATA which is again based on the REQUIRE file. CIBot is based on the General registry so project (or manifest) files will not affect the reverse dependency testing. Once we transition to releasing directly to General, see https://github.com/JuliaLang/Pkg.jl/issues/849, the REQUIRE file will no longer be needed.

Feel free to ask any questions.

39 Likes
2

Does it mean ENV["PYTHON"] is set to Conda?

3

Does it mean ENV["PYTHON"] is set to Conda ?

Almost. It’s set to ENV["PYTHON"]="" which defaults to Conda.

4

I see. Thanks.

5

It seems to me like a lot of (all?) PyCall-backed packages just don’t work when ENV["PYTHON"] != "". This has always been a minor frustration for me. If even METADATA CI is defaulting to ENV["PYTHON"] = "" now, should the default in PyCall just be changed?

3 Likes
6

What exactly is the “minor frustration”? That PyCall.jl does not auto-magically install Python packages when not using Conda.jl?

7

I guess, if that’s what’s happening. I’m not familiar with PyCall internals. All I know is that when e.g. using SymPy in a test on Travis, I need to set ENV["PYTHON"]="" or it doesn’t work.

8

Just wanted to say, thanks so much for writing all this up! I think the current transitional state of METADATA/General has been a source of confusion for post-1.0 Julia newcomers, and having all this info in one place really helps to clarify the situation.

7 Likes
9

I guess in principle using Conda.jl can be the default in PyCall.jl for Linux as well. Maybe open an issue in github? There is always a trade-off and I’m not sure what’s the best. Though @stevengj may already have some reasons to avoid it.

10

Originally PyCall defaulted to whatever Python you had installed, and installed Conda as a fallback. On Mac and Windows we switched to defaulting to Conda because we encountered broken Python environments extremely often on those platforms.

We continued to default to the system python (now python3) on other Unix platforms, partly to be conservative, partly because GNU/Linux systems tend to have a working up-to-date Python, and partly because non-Mac Unix users have different expectations and might be upset if we defaulted to installing our own Anaconda Python distro rather than using a python in their PATH.

If there is a clamor by Ubuntu users for us to default to installing our own Python, we can easily switch to using Conda by default, but I haven’t heard that until now. Travis testing is another matter, but normally I would suggest testing with the Ubuntu packages of sympy etcetera in order to mirror the setup of a typical user.

3 Likes
11

One benefit of Conda.jl being default in Linux is that we can be sure that the CI setup mirrors typical users’ setup (:= no custom ENV["PYTHON"] configuration) even across Linux distros.

(Not that I prefer this default personally. But it’s already configurable so I won’t care much.)

12

This is my sentiment exactly.

13

Thanks for writing this up, and thanks for implementing the new automated system. I think it is quite intuitive - I managed to infer a lot of the points you make above just by messing around with new releases on a package for an hour or so a few days back.

14

I have a question about CIBot in the specific case of the StaticArrays 0.9.0 tag. Here’s the CIBot output. Why didn’t CIBot test RigidBodyDynamics.jl (even though it isn’t marked as untestable)? In general, I would think that StaticArrays has many more reverse dependencies.

1 Like
15

That is good question. I forgot to mention that in the top post. We currently limit the number of tested package to ten to avoid tests running for too long.Personally, I think it would be great to set that limit higher since a lot of reverse dependencies reflects that many people rely on a package so it’s worth spending the extra time and compute verifying that nothing breaks. We might gradually increase the number of packages tested to see how it goes.

6 Likes
16

Hi Andreas, thanks for all the great work on this! It’s been working great.

When trying to tag a new version of HyperDualNumbers.jl we ran into some issues. I have some suggestions/questions related to JuliaCIBot:

  1. Should an update request be marked as fail vs. just letting the failed dependencies know about this? Or maybe just when it is updating the major version number?

  2. Should dependent packages within a Github organization have preference to be tested? Assuming being part of the same Github organization could be established for this purpose.

  3. Or is it a good idea to add a section for this to Project.toml? That way dependent packages can create a PR requesting to be added to such a section.

  4. Finally, it would be really nice to be able to publish (or include in the docs) the JuliaCIBot page. Is it only on AWS?

Rob

17
  1. Should an update request be marked as fail vs. just letting the failed dependencies know about this?

In the future, we could fine tune this but most breakages seem to be unintentional in which case the failure status is useful.

  1. Should dependent packages within a Github organization have preference to be tested? Assuming being part of the same Github organization could be established for this purpose.
  2. Or is it a good idea to add a section for this to Project.toml? That way dependent packages can create a PR requesting to be added to such a section.

I don’t understand what you are asking here. Could you elaborate?

  1. Finally, it would be really nice to be able to publish (or include in the docs) the JuliaCIBot page. Is it only on AWS?

The reporting could definitely be improved. However, we’ll have to restructure and rewrite CIBot when migrating to the new non-METADATA based registration system so it’s unlikely that we’ll make changes before the new system is up and running.

18

Hi Andreas, thanks for your response.

It’s great to have visibility into which dependent packages will fail after an update (which is shown on JuliaCIBot page). Items 2 and 3 are ideas to provide developers of packages some control and focus over when a package update is marked to have failed. Instead of testing 10 (or more in the future) randomly selected packages, the tested packages are part of a github organization or specified in a Project.toml section.

I do realize a lot of this is still in flux!

1 Like
19

Update: 2 January 2019

Because of problems caused by updates to Azure, we are currently unable to deploy worker nodes for CIBot. Hence, CIBot verification of METADATA PRs have been suspended and PRs will be merged if just the METADATA tests pass.

We currently don’t know when CIBot will be up and running again. Hence, for the time being, it is adviced that package authors are cautious about breaking changes.

1 Like
20

I don’t know if this has been discussed anywhere (I’d imagine it probably has), but what is the plan for supporting automatic METADATA/General merges for packages that aren’t hosted on Github? I host most of my packages on Gitlab, and plan to host some more on a different non-Github service in the near future. It would be nice if these packages could also benefit from a streamlined, automated METADATA/General release process.

1 Like