We are experimenting with some changes to the storage backend of the package servers[1]. These changes are currently live on https://staging.pkg.julialang.org. It would be great if some of you could beta test the new server configuration and report back either here or in the #pkg-dev channel on Slack. I have alpha-tested this server myself for a while with no issues. Edit: The test is successful if you don’t notice any difference from before.
The simplest way to configure Pkg to use the staging server is to add
to your Julia startup file (~/.julia/config/startup.jl). Alternatively, you can of course set this environment variable in your shell startup script if you prefer, for example
Package servers are the servers where Pkg, Julia’s package manager, downloads package registries, package, and artifacts from. By default https://pkg.julialang.org is used, which redirects to a server closer to your location. You can visit https://pkg.julialang.org with your browser to see where you are redirected (for me and others in Europe it will be https://eu-central.pkg.julialang.org for example). ↩︎
Just wanted to post an update with some more details about this change.
As I briefly mentioned in the first post, Pkg by default connects to https://pkg.julialang.org which simply redirects everything to a regional server. For me, located in Europe, this will be https://eu-central.pkg.julialang.org, for example. Up until now all resources (dynamic content such as current registry and static content such as registry/package/artifact tarballs) have been served/proxied through this caching server.
We are now trying out a different caching storage server hosted at https://storage.julialang.net (note .net instead of .org) backed by Cloudflare R2. (Using another top level domain was unfortunately necessary but we are looking into the possibility of using .org for this server too.) In order to reduce network traffic (and thus reduce cost for the JuliaLang organization) the regional servers redirects (instead of proxies) requests for static content to https://storage.julialang.net. This has been live on https://staging.pkg.julialang.org since the original post. For the majority of users this will be an invisible change, but if you are behind a proxy or similar you might have to allow requests to the new domain too.
Last week this change was rolled out to the https://us-east.pkg.julialang.org region. This is a very busy region (in part because lots of traffic from CI services end up here) and it wasn’t the initial plan to deploy here first. However, the existing infrastructure for this region was under a lot of pressure already and moving to the new system was the simplest way forward. I completely forgot to post an update about this here, sorry.
Luckily, as far as I know, the new system seems to work nicely. It did reveal a bug in HTTP.jl, some servers with old certificates, and (rightfully) some questions from observant sysadmins.
Please post here or in the #pkg-dev if you have any questions or problems related to this.
Whatever was done to the nominal Package manager storage system. My company firewalls are now blocking it which is cutting me off from the registry. Trying to regain access.
Only the subdomain storage.julialang.net is currently used so julialang.net is not assigned to anything, and yes that specific URL is a 404. This is not meant to be visited in your browser, it’s infrastructure. It is still a valid domain, just not a URL you can see. If you visit the long julialang.net URL that you posted above then it will initiate a download if you aren’t behind a firewall that blocks it.
Again: ask your sysadmin to allow access to at least the storage.julialang.net domain and ideally all subdomains of julialang.net (in case we use more subdomains in the future). If they do this then the new setup will work, if they don’t then it won’t.
Could we create a page on julialang.org or maybe juliahub.com that states what domain names are officially associated with Julia? I’ve been fielding questions via email from system administrators and network operation managers for contact@julialang.org. The best I can do is point them here.
Hello, it is now impossible to install or update julia packages. Indeed, to install packages, we had to open gatekeepers to reach github.com, github.io, julia-vscode.org and julialang.org. Now it seems that we have to open new gatekeepers since we have now to reach new adress such as (?) objects.githubusercontent.com. But it seems that this isn’t enough and that we need to open others, such as storage.julialang.net. Could you give me the exact list of all the sites we need to reach to update Julia Packages ? Note that if this list is dynamic, we won’t be able to manage it. Indeed, we’re under strict security constraints and can’t open gateways easily. Thank you in advance for the informations.
Access to GitHub domains is only needed if you want to install unregistered packages that are hosted on GitHub. You are not required to do that in any way. The fact that Pkg can get things from GitHub for you is a convenience. If GitHub now uses new domains like objects.githubusercontent.com that is not under our control and you can feel free to ignore it, just don’t expect direct installs from GitHub to work. If tomorrow they add some other subdomain, guess what? You’ll have to add that.
I don’t know about the julia-vscode.org domain. Presumably that is for the Julia VS Code extension. If you don’t use that then you don’t need this. I also think it’s just a website? Not sure why it would be blocked in the first place.
That leaves exactly two domains that are needed for normal Pkg operations to work. There used to be one. Now, for technical reasons beyond our control, there are two. Sorry for the inconvenience. This was not done to annoy you. Believe me if there was a way to avoid having to have this conversation i would have preferred it.
this might not be completely true if one is using JuliaUp, at least it tried to grab something from Github the other day when launched for the first time. (I hit this problem because I was on a weird system without ~/.git access). It’s possible it might continue to work somehow, but just want to put this out here for the completeness of the discussion.
Okay, so as far as I can tell, when you first install Juliaup, it downloads the Juliaup binaries from a URL of the form https://github.com/JuliaLang/juliaup/releases/latest/download/*.[1]
Therefore, the first time that a user installs Juliaup, they need to be able to make network requests to the github.com domain.
So, we should probably add a note (to the “Official Domains” section that informs users that they need to be able to make network requests to the github.com domain, but only if they are using Juliaup.
If the user is not using Juliaup, then they do not need access to the github.com domain.
I’m not sure if Juliaup also makes requests to github.com when updating Juliaup itself, or if access to github.com is only needed when installing Juliaup for the first time. @davidanthoff Do you know the answer to this question?
For Juliaup, as far as I can tell, there are no other domains needed by Juliaup, except for github.com (as mentioned in my previous comment), and of course *.julialang.org and *.julialang.net.
If someone finds an additional domain (or subdomain) that is needed for Juliaup to work, please let me know.