# Pkg downtime incident
**URL:** https://discourse.julialang.org/t/pkg-downtime-incident/44288
**Category:** Announcements
**Tags:** package
**Created:** [August 4, 2020, 9:58pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288 "2020-08-04T21:58:19Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Keno](https://discourse.julialang.org/u/Keno)
#### Post date: [August 4, 2020, 9:58pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/1 "2020-08-04T21:58:19Z")
Earlier today, several users started seeing issues installing packages. This post seeks to collect all the information related to this incident.
# Impact
The issue caused installation of incorrect versions (latest master when a prior version was requested) of packages.
- Versions of Julia prior to 1.4 will silently install the wrong version
- Windows versions of Julia 1.4.x will also silently install the wrong version
- Non-Windows versions of Julia 1.4.x will issue a warning and fall back to git to obtain the correct version
- Julia 1.5 is unaffected when using the pkg server (which is the default), otherwise matches 1.4 behavior
The issue has since been mitigated in the registry, so you were only affected if you were attempting package operations on an affected version between approximately 2pm Eastern and 3:43pm Eastern when the mitigation went into effect.
# Symptoms
Installing the wrong version of a Julia package can cause incorrect behavior in several different ways. Perhaps the most common will be inscrutable package dependency errors, but more subtle behaviors are possible. If you performed a package operation today, you may want to see the mitigation section below as a precaution.
# Mitigation
If an incorrect package version was installed, it will be locally cached until removed. As such, if you believe you were affected, it is advisable to clear your package cache by deleting `.julia/packages`. Note that your list of installed packages will not be affected and you may re-download all installed packages in your current environment by using `Pkg.instantiate()`.
# Root cause
The root cause of this change was an unannounced serverside change by GitHub, which broke download of tarballs by git-tree-hash, e.g. previously [https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6](https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6) would give the tarball for that tree-hash, while it now gives the tarball for master instead. We do not yet know whether this change was intentional or not. The reason this change broke Pkg is that Pkg includes a heuristic where it will use the tarball download feature instead of a full git checkout as faster way to download a requested version (since it no longer needs to download the full repository with all its history). This was special cased for [github.com](http://github.com) and does not affect packages hosted elsewhere (though the vast majority of packages are currently hosted on GitHub).
# Registry workaround
The above mentioned workaround was [https://github.com/JuliaRegistries/General/pull/18991/files](https://github.com/JuliaRegistries/General/pull/18991/files), which changes the URL for all registered packages from `github.com` to `GitHub.com`. This breaks above mentioned heuristic and will force older versions of Julia to fall back to a full git checkout instead. This method is slower, but should yield the correct package version. Note that Julia 1.5+ is unaffected and downloads via the Pkg server will continue to be fast.
# Additional considerations/General registry updates paused
We have contacted GitHub to find out whether this change was intentional and is likely to persist. If so, we will need to update Registrator and the validation CI to force packages registered at GitHub to use the same `GitHub.com` workaround we manually applied to the registry. If not, the workaround will be reverted as soon as GitHub restores the original behavior (to get back to faster package download speeds on older versions). In the meantime changes (new packages/version bumps) to the General registry are paused. They will be resumed once either of the two options have been completed.
# Future considerations
As noted, Julia versions 1.5+ are not affected due to the Pkg server work (which was partly motivated by a desire to avoid incidents like this once). However, such Julia versions will still fall back to raw GitHub downloads if the package server is unavailable for some reason (broken, blocked by corporate firewall, we forgot to pay our bills, etc.). In the near future, the validation currently present on non-Windows versions, will be extended to Windows version, such that even with a broken package server, the fall back path would itself fallback to Git if it is being served incorrect tarballs (the same verification will of course extend to the package server also). This change has been planned for some time and the requisite support is already available in Tar.jl, but has not yet been wired up in Pkg.
---
### Author:  [@alhirzel](https://discourse.julialang.org/u/alhirzel)
#### Post date: [August 4, 2020, 11:05pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/2 "2020-08-04T23:05:30Z")
> [@Keno](#):
>
> which changes the URL for all registered packages from `github.com` to `GitHub.com` . This breaks above mentioned heuristic
This is “clever”. 🙂
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [August 4, 2020, 11:29pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/3 "2020-08-04T23:29:37Z")
Fight failure with failure, as they say.
---
### Author:  [@anon92994695](https://discourse.julialang.org/u/anon92994695)
#### Post date: [August 5, 2020, 12:26am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/4 "2020-08-05T00:26:17Z")
Is the following related to this?
```nohighlight
Warning: Due to a previously reported error, the running code does not match saved version for the following files:
│
│ /home/usernamehere/.julia/packages/Flux/IjMZL/src/layers/basic.jl
│
│ Use Revise.errors() to report errors again.
```
Note, I’m not running Revise at all(must be a depend somewhere)…
---
### Author:  [@Keno](https://discourse.julialang.org/u/Keno)
#### Post date: [August 5, 2020, 12:27am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/5 "2020-08-05T00:27:37Z")
> [@anon92994695](#):
>
> Is the following related to this?
Possibly, but unlikely.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [August 5, 2020, 12:33am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/6 "2020-08-05T00:33:41Z")
> [@Keno](#):
>
> Pkg includes a heuristic where it will use the tarball download feature instead of a full git checkout as faster way to download a requested version (since it no longer needs to download the full repository with all its history)
Is it possible to switch to a shallow clone? I imagine this would be a bit slower than a tarball, but substantially better than cloning a full history.
---
### Author:  [@Keno](https://discourse.julialang.org/u/Keno)
#### Post date: [August 5, 2020, 12:34am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/7 "2020-08-05T00:34:33Z")
> [@kevbonham](#):
>
> Is it possible to switch to a shallow clone? I imagine this would be a bit slower than a tarball, but substantially better than cloning a full history.
GitHub requests people don’t do this because the full clone path is special cased while the shallow clone case uses a slow path that causes lots of CPU load.
---
### Author:  [@PeterB](https://discourse.julialang.org/u/PeterB)
#### Post date: [August 5, 2020, 12:56am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/8 "2020-08-05T00:56:28Z")
“2pm Eastern to 3.43 Eastern”. I appreciate this email was probably rushed out ASAP, but does that mean Eastern Standard Time (UT-5) or Eastern Daylight Time (UT-4)?
Then again, being on UT+9.5 myself, I was fast asleep either way 🙂
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [August 5, 2020, 12:57am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/9 "2020-08-05T00:57:28Z")
The problem is that we cannot change the Julia client in any way because people already have it and we can’t change code on their computers. Otherwise we could just remove this behavior or allow it to be overridden by the registry.
---
### Author:  [@Keno](https://discourse.julialang.org/u/Keno)
#### Post date: [August 5, 2020, 12:58am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/10 "2020-08-05T00:58:15Z")
It means the time that people on the east coast of the US say on their clocks 😉 - in this case Eastern Daylight Time.
---
### Author:  [@e3c6](https://discourse.julialang.org/u/e3c6)
#### Post date: [August 5, 2020, 5:37am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/11 "2020-08-05T05:37:06Z")
I am on Julia 1.5. How do I know I am using the new Pkg protocol? Is there a warning printed if that fails and it falls back to the old Github downloads?
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [August 5, 2020, 6:08am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/12 "2020-08-05T06:08:40Z")
Has github been made aware of this? Since its a publicly accessible API, surely this must have been unintentional?
The original URL is being redirected to
```julia
https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6
```
Seems like they either want to deprecate tarball downloads or do something else with them since they’re calling them `legacy.tar.gz`?
Funnily enough, this sudden change in behaviour goes directly against their [recently published roadmap](https://github.com/github/roadmap) on how they want to roll out new features…
---
### Author:  [@Keno](https://discourse.julialang.org/u/Keno)
#### Post date: [August 5, 2020, 6:13am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/13 "2020-08-05T06:13:35Z")
Yes, GitHub is aware. Our present understanding is that they’ll be rolling back this change, but we’re awaiting explicit confirmation to that extent
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [August 5, 2020, 7:34am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/14 "2020-08-05T07:34:44Z")
> [@Sukera](#):
>
> The original URL is being redirected to
>
> ```julia
> https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/2d94286a9c2f52c63
>
> ```
Actually, I think that means it is fixed now on GitHub’s side. Previously it resolved to `https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/master` but now it has the tree hash in the URL and the downloaded file seems to contain the correct files.
---
### Author:  [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)
#### Post date: [August 5, 2020, 10:51am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/15 "2020-08-05T10:51:13Z")
The workarounds have been reverted and everything should be back to normal now.
---
### Author:  [@stev47](https://discourse.julialang.org/u/stev47)
#### Post date: [August 5, 2020, 10:52am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/16 "2020-08-05T10:52:08Z")
As long as the git server supports it (GitHub seems to), you can shallow clone a commit through
```julia
mkdir && cd
git init
git remote add origin
git fetch --depth 1 origin [
git checkout FETCH_HEAD
```
Might be more viable that to handcode tarball urls for different git servers.
---
]
### Author:  [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)
#### Post date: [August 5, 2020, 11:02am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/17 "2020-08-05T11:02:39Z")
> [@Pkg downtime incident](https://discourse.julialang.org/t/pkg-downtime-incident/44288/7):
>
> GitHub requests people don’t do this because the full clone path is special cased while the shallow clone case uses a slow path that causes lots of CPU load.
---
### Author:  [@essenciary](https://discourse.julialang.org/u/essenciary)
#### Post date: [August 5, 2020, 11:36am UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/18 "2020-08-05T11:36:14Z")
Thanks for your quick response and hard work to mitigate this.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [August 5, 2020, 2:09pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/19 "2020-08-05T14:09:40Z")
Update on this:
- GitHub has rolled back the change, so the API for downloading tarballs by tree hash is no longer serving incorrect tarballs.
- The registry workaround hack has been reverted so people using older Julia versions will get (correct) tarball downloads again.
- Auto-merge on the General registry has been turned back on.
In short: **everything is back to normal now**.
GitHub was very responsive about this and we’re very grateful to them for that. These kinds of things happen when you run a service and the best that one can do is to react quickly when something goes wrong.
The major lesson here is that the Pkg client should not rely on baked in knowledge of 3rd party APIs in a way that cannot be overridden remotely by making metadata changes to the registry. Of course, had GitHub simply discontinued this API and returned a 404, then everything would have been fine, since Pkg would try it, fail and fall back to cloning the package repo. What caused the problem was that the API seemed to continue working while returning the _wrong_ content, which is a failure mode that’s pretty hard to anticipate. Fortunately, with the Pkg protocol in 1.5, we no longer rely on 3rd party APIs like this.
---
### Author:  [@hendri54](https://discourse.julialang.org/u/hendri54)
#### Post date: [August 5, 2020, 3:40pm UTC](https://discourse.julialang.org/t/pkg-downtime-incident/44288/20 "2020-08-05T15:40:59Z")
`Pkg.update` shows where it downloads the registry. In my case, the switchover to the new protocol was not automatic. See
> [@Pkg.update in Julia 1.5 uses github](https://discourse.julialang.org/t/pkg-update-in-julia-1-5-uses-github/44331):
>
> When I Pkg.update in Julia 1.5, Pkg seems to still pull from github: (@v1.5) pkg\> up Updating registry at `~/.julia/registries/General` Updating git-repo `https://github.com/JuliaRegistries/General` Updating registry at `~/.julia/registries/registryLH` Updating git-repo `https://github.com/hendri54/registryLH` No Changes to `~/.julia/environments/v1.5/Project.toml` No Changes to `~/.julia/environments/v1.5/Manifest.toml` Is there a setting that I need to change to switch to the new…
[Next page](https://discourse.julialang.org/t/pkg-downtime-incident/44288.md?page=2)