Around 2023-03-06T08:16:00Z the DNS resolver for the package servers went offline.
As a temporary measure we are resolving the DNS names for the pkgservers directly. This minimizes impact while we are working on resolving the core issue.
Impacted users can set the environment variable
JULIA_PKG_SERVER="" to bypass the pkgservers.
Please remember to unset the environment variable after the outage has been resolved.
At 2023-03-06T15:40:00Z the load balancing DNS server resumed functionality and full service was restored.
This outage highlighted a single point of failure in our pkgserver infrastructure. We will add fallback servers to avoid future outages.
- 2023-03-06T08:16:00Z Outage reported on Slack.
- 2023-03-06T09:51:00Z Investigation started.
- 2023-03-06T10:34:00Z First report update.
- 2023-03-06T10:51:00Z First mitigation attempt for us-east
- 2023-03-06T10:56:00Z Outage mitigated
- 2023-03-06T15:40:00Z Outage resolved
Thank you to all the community members who quickly reported this issue! Special thanks to @fredrikekre, Pradeep, @aviks who collaborated to mitigate this issue quickly. Thanks to @staticfloat for the final resolution of the issue and his continued efforts in maintaining the JuliaLang infrastructure.
The JuliaLang open-source infrastructure is maintained by volunteers and it was greet to see folks come together over this.