Changes to PkgServers storage backend

We are experimenting with some changes to the storage backend of the package servers[1]. These changes are currently live on https://staging.pkg.julialang.org. It would be great if some of you could beta test the new server configuration and report back either here or in the #pkg-dev channel on Slack. I have alpha-tested this server myself for a while with no issues. Edit: The test is successful if you don’t notice any difference from before.

The simplest way to configure Pkg to use the staging server is to add

ENV["JULIA_PKG_SERVER"] = "https://staging.pkg.julialang.org"

to your Julia startup file (~/.julia/config/startup.jl). Alternatively, you can of course set this environment variable in your shell startup script if you prefer, for example

export JULIA_PKG_SERVER=https://staging.pkg.julialang.org

in .bashrc.

Thanks!


  1. Package servers are the servers where Pkg, Julia’s package manager, downloads package registries, package, and artifacts from. By default https://pkg.julialang.org is used, which redirects to a server closer to your location. You can visit https://pkg.julialang.org with your browser to see where you are redirected (for me and others in Europe it will be https://eu-central.pkg.julialang.org for example). ↩︎

3 Likes

Should I notice any differences when I do this?

Hopefully not.

1 Like

Just wanted to post an update with some more details about this change.

As I briefly mentioned in the first post, Pkg by default connects to https://pkg.julialang.org which simply redirects everything to a regional server. For me, located in Europe, this will be https://eu-central.pkg.julialang.org, for example. Up until now all resources (dynamic content such as current registry and static content such as registry/package/artifact tarballs) have been served/proxied through this caching server.

We are now trying out a different caching storage server hosted at https://storage.julialang.net (note .net instead of .org) backed by Cloudflare R2. (Using another top level domain was unfortunately necessary but we are looking into the possibility of using .org for this server too.) In order to reduce network traffic (and thus reduce cost for the JuliaLang organization) the regional servers redirects (instead of proxies) requests for static content to https://storage.julialang.net. This has been live on https://staging.pkg.julialang.org since the original post. For the majority of users this will be an invisible change, but if you are behind a proxy or similar you might have to allow requests to the new domain too.

Last week this change was rolled out to the https://us-east.pkg.julialang.org region. This is a very busy region (in part because lots of traffic from CI services end up here) and it wasn’t the initial plan to deploy here first. However, the existing infrastructure for this region was under a lot of pressure already and moving to the new system was the simplest way forward. I completely forgot to post an update about this here, sorry.

Luckily, as far as I know, the new system seems to work nicely. It did reveal a bug in HTTP.jl, some servers with old certificates, and (rightfully) some questions from observant sysadmins.

Please post here or in the #pkg-dev if you have any questions or problems related to this.

Thanks!

8 Likes