Changes to PkgServers storage backend

Just wanted to post an update with some more details about this change.

As I briefly mentioned in the first post, Pkg by default connects to https://pkg.julialang.org which simply redirects everything to a regional server. For me, located in Europe, this will be https://eu-central.pkg.julialang.org, for example. Up until now all resources (dynamic content such as current registry and static content such as registry/package/artifact tarballs) have been served/proxied through this caching server.

We are now trying out a different caching storage server hosted at https://storage.julialang.net (note .net instead of .org) backed by Cloudflare R2. (Using another top level domain was unfortunately necessary but we are looking into the possibility of using .org for this server too.) In order to reduce network traffic (and thus reduce cost for the JuliaLang organization) the regional servers redirects (instead of proxies) requests for static content to https://storage.julialang.net. This has been live on https://staging.pkg.julialang.org since the original post. For the majority of users this will be an invisible change, but if you are behind a proxy or similar you might have to allow requests to the new domain too.

Last week this change was rolled out to the https://us-east.pkg.julialang.org region. This is a very busy region (in part because lots of traffic from CI services end up here) and it wasn’t the initial plan to deploy here first. However, the existing infrastructure for this region was under a lot of pressure already and moving to the new system was the simplest way forward. I completely forgot to post an update about this here, sorry.

Luckily, as far as I know, the new system seems to work nicely. It did reveal a bug in HTTP.jl, some servers with old certificates, and (rightfully) some questions from observant sysadmins.

Please post here or in the #pkg-dev if you have any questions or problems related to this.

Thanks!

10 Likes