Outage: Julia package server offline [resolved]

vchuravy
Mon, 06 Mar 2023 10:40:09 +0000
Announcements

Status: Resolved

Description

Around Mon, Mar 6, 2023 8:16 AM UTC the DNS resolver for the package servers went offline.

Mitigation

As a temporary measure we are resolving the DNS names for the pkgservers directly. This minimizes impact while we are working on resolving the core issue.
Outdated (click for more details)
Resolution

At Mon, Mar 6, 2023 3:40 PM UTC the load balancing DNS server resumed functionality and full service was restored.

Next steps

This outage highlighted a single point of failure in our pkgserver infrastructure. We will add fallback servers to avoid future outages.

Events

Mon, Mar 6, 2023 8:16 AM UTC Outage reported on Slack.

Mon, Mar 6, 2023 9:51 AM UTC Investigation started.

Mon, Mar 6, 2023 10:34 AM UTC First report update.

Mon, Mar 6, 2023 10:51 AM UTC First mitigation attempt for us-east

Mon, Mar 6, 2023 10:56 AM UTC Outage mitigated

Mon, Mar 6, 2023 3:40 PM UTC Outage resolved

Conclusion

Thank you to all the community members who quickly reported this issue! Special thanks to @fredrikekre, Pradeep, @aviks who collaborated to mitigate this issue quickly. Thanks to @staticfloat for the final resolution of the issue and his continued efforts in maintaining the JuliaLang infrastructure.

The JuliaLang open-source infrastructure is maintained by volunteers and it was greet to see folks come together over this.

2 posts - 2 participants

Read full topic