Pkg downtime incident

This is “clever”. :slight_smile:

4 Likes

Fight failure with failure, as they say.

19 Likes

Is the following related to this?

 Warning: Due to a previously reported error, the running code does not match saved version for the following files:
│ 
│   /home/usernamehere/.julia/packages/Flux/IjMZL/src/layers/basic.jl
│ 
│ Use Revise.errors() to report errors again.

Note, I’m not running Revise at all(must be a depend somewhere)…

Possibly, but unlikely.

1 Like

Is it possible to switch to a shallow clone? I imagine this would be a bit slower than a tarball, but substantially better than cloning a full history.

1 Like

GitHub requests people don’t do this because the full clone path is special cased while the shallow clone case uses a slow path that causes lots of CPU load.

4 Likes

“2pm Eastern to 3.43 Eastern”. I appreciate this email was probably rushed out ASAP, but does that mean Eastern Standard Time (UT-5) or Eastern Daylight Time (UT-4)?

Then again, being on UT+9.5 myself, I was fast asleep either way :slight_smile:

The problem is that we cannot change the Julia client in any way because people already have it and we can’t change code on their computers. Otherwise we could just remove this behavior or allow it to be overridden by the registry.

1 Like

It means the time that people on the east coast of the US say on their clocks :wink: - in this case Eastern Daylight Time.

3 Likes

I am on Julia 1.5. How do I know I am using the new Pkg protocol? Is there a warning printed if that fails and it falls back to the old Github downloads?

1 Like

Has github been made aware of this? Since its a publicly accessible API, surely this must have been unintentional?

The original URL is being redirected to

https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6

Seems like they either want to deprecate tarball downloads or do something else with them since they’re calling them legacy.tar.gz?

Funnily enough, this sudden change in behaviour goes directly against their recently published roadmap on how they want to roll out new features…

1 Like

Yes, GitHub is aware. Our present understanding is that they’ll be rolling back this change, but we’re awaiting explicit confirmation to that extent

7 Likes

Actually, I think that means it is fixed now on GitHub’s side. Previously it resolved to https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/master but now it has the tree hash in the URL and the downloaded file seems to contain the correct files.

7 Likes

The workarounds have been reverted and everything should be back to normal now.

7 Likes

As long as the git server supports it (GitHub seems to), you can shallow clone a commit through

mkdir <repo> && cd <repo>
git init
git remote add origin <url>
git fetch --depth 1 origin <ref/sha>
git checkout FETCH_HEAD

Might be more viable that to handcode tarball urls for different git servers.

4 Likes

Thanks for your quick response and hard work to mitigate this.

13 Likes

Update on this:

  • GitHub has rolled back the change, so the API for downloading tarballs by tree hash is no longer serving incorrect tarballs.
  • The registry workaround hack has been reverted so people using older Julia versions will get (correct) tarball downloads again.
  • Auto-merge on the General registry has been turned back on.

In short: everything is back to normal now.

GitHub was very responsive about this and we’re very grateful to them for that. These kinds of things happen when you run a service and the best that one can do is to react quickly when something goes wrong.

The major lesson here is that the Pkg client should not rely on baked in knowledge of 3rd party APIs in a way that cannot be overridden remotely by making metadata changes to the registry. Of course, had GitHub simply discontinued this API and returned a 404, then everything would have been fine, since Pkg would try it, fail and fall back to cloning the package repo. What caused the problem was that the API seemed to continue working while returning the wrong content, which is a failure mode that’s pretty hard to anticipate. Fortunately, with the Pkg protocol in 1.5, we no longer rely on 3rd party APIs like this.

34 Likes

Pkg.update shows where it downloads the registry. In my case, the switchover to the new protocol was not automatic. See

3 Likes

People who already have a git cloned ~/.julia/registries/General will continue to use git to update the registry, but only the registry. This should not make a significant difference and has no impact on this incident since registry updates were not broken. If you want to use Pkg protocol for General registry updates, you can delete ~/.julia/registries/General.

5 Likes