Update on this:
- GitHub has rolled back the change, so the API for downloading tarballs by tree hash is no longer serving incorrect tarballs.
- The registry workaround hack has been reverted so people using older Julia versions will get (correct) tarball downloads again.
- Auto-merge on the General registry has been turned back on.
In short: everything is back to normal now.
GitHub was very responsive about this and we’re very grateful to them for that. These kinds of things happen when you run a service and the best that one can do is to react quickly when something goes wrong.
The major lesson here is that the Pkg client should not rely on baked in knowledge of 3rd party APIs in a way that cannot be overridden remotely by making metadata changes to the registry. Of course, had GitHub simply discontinued this API and returned a 404, then everything would have been fine, since Pkg would try it, fail and fall back to cloning the package repo. What caused the problem was that the API seemed to continue working while returning the wrong content, which is a failure mode that’s pretty hard to anticipate. Fortunately, with the Pkg protocol in 1.5, we no longer rely on 3rd party APIs like this.