# Changes to PkgServers storage backend

**URL:** <https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378>\
**Category:** Community\
**Created:** [April 1, 2024, 4:02pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378 "2024-04-01T16:02:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [April 1, 2024, 4:02pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/1 "2024-04-01T16:02:03Z")

</div>

We are experimenting with some changes to the storage backend of the package servers\[1\]. These changes are currently live on [https://staging.pkg.julialang.org](https://staging.pkg.julialang.org). It would be great if some of you could beta test the new server configuration and report back either here or in the [#pkg-dev](https://app.slack.com/client/T68168MUP/CBF3Z1D7V) channel on Slack. I have alpha-tested this server myself for a while with no issues. Edit: The test is successful if you _don’t_ notice any difference from before.

The simplest way to configure Pkg to use the staging server is to add

```julia
ENV["JULIA_PKG_SERVER"] = "https://staging.pkg.julialang.org"

```

to your Julia startup file (`~/.julia/config/startup.jl`). Alternatively, you can of course set this environment variable in your shell startup script if you prefer, for example

```julia
export JULIA_PKG_SERVER=https://staging.pkg.julialang.org

```

in `.bashrc`.

Thanks!

* * *

1. Package servers are the servers where Pkg, Julia’s package manager, downloads package registries, package, and artifacts from. By default [https://pkg.julialang.org](https://pkg.julialang.org) is used, which redirects to a server closer to your location. You can visit [https://pkg.julialang.org](https://pkg.julialang.org) with your browser to see where you are redirected (for me and others in Europe it will be [https://eu-central.pkg.julialang.org](https://eu-central.pkg.julialang.org) for example).

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [April 1, 2024, 4:51pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/2 "2024-04-01T16:51:17Z")

</div>

Should I notice any differences when I do this?

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [April 1, 2024, 5:06pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/3 "2024-04-01T17:06:31Z")

</div>

Hopefully not.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [April 15, 2024, 9:52pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/4 "2024-04-15T21:52:01Z")

</div>

Just wanted to post an update with some more details about this change.

As I briefly mentioned in the first post, Pkg by default connects to [https://pkg.julialang.org](https://pkg.julialang.org) which simply redirects everything to a regional server. For me, located in Europe, this will be [https://eu-central.pkg.julialang.org](https://eu-central.pkg.julialang.org), for example. Up until now all resources (dynamic content such as current registry and static content such as registry/package/artifact tarballs) have been served/proxied through this caching server.

We are now trying out a different caching storage server hosted at [https://storage.julialang.net](https://storage.julialang.net) (note `.net` instead of `.org`) backed by Cloudflare R2. (Using another top level domain was unfortunately necessary but we are looking into the possibility of using `.org` for this server too.) In order to reduce network traffic (and thus reduce cost for the JuliaLang organization) the regional servers redirects (instead of proxies) requests for static content to [https://storage.julialang.net](https://storage.julialang.net). This has been live on [https://staging.pkg.julialang.org](https://staging.pkg.julialang.org) since the original post. For the majority of users this will be an invisible change, but if you are behind a proxy or similar you might have to allow requests to the new domain too.

Last week this change was rolled out to the [https://us-east.pkg.julialang.org](https://us-east.pkg.julialang.org) region. This is a very busy region (in part because lots of traffic from CI services end up here) and it wasn’t the initial plan to deploy here first. However, the existing infrastructure for this region was under a lot of pressure already and moving to the new system was the simplest way forward. I completely forgot to post an update about this here, sorry.

Luckily, as far as I know, the new system seems to work nicely. It did reveal a [bug in HTTP.jl](https://github.com/JuliaWeb/HTTP.jl/issues/1165), some [servers with old certificates](https://discourse.julialang.org/t/bad-cert-when-accessing-the-general-registry/112738), and (rightfully) some questions from observant sysadmins.

Please post here or in the [#pkg-dev](https://app.slack.com/client/T68168MUP/CBF3Z1D7V) if you have any questions or problems related to this.

Thanks!

---

<div class="post-metadata">

**Author:** ![Allan\_Baker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/allan_baker/32/42645_2.png) [@Allan\_Baker](https://discourse.julialang.org/u/Allan_Baker)\
**Post date:** [April 25, 2024, 1:54pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/5 "2024-04-25T13:54:26Z")

</div>

Whatever was done to the nominal Package manager storage system. My company firewalls are now blocking it which is cutting me off from the registry. Trying to regain access.

[https://storage.julialang.net/registry/23338594-aafe-5451-b93e-139f81909106/f8c090c98cada1c7542aa9c09c3e75e0e57a3121.tar.gz](https://storage.julialang.net/registry/23338594-aafe-5451-b93e-139f81909106/f8c090c98cada1c7542aa9c09c3e75e0e57a3121.tar.gz)

---

<div class="post-metadata">

**Author:** ![Allan\_Baker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/allan_baker/32/42645_2.png) [@Allan\_Baker](https://discourse.julialang.org/u/Allan_Baker)\
**Post date:** [April 25, 2024, 2:38pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/6 "2024-04-25T14:38:31Z")

</div>

It looks like I have to do this to avoid a firewall error.

```julia
(@v1.10) pkg> registry add https://github.com/JuliaRegistries/General.git
(https://github.com/JuliaRegistries/General.git)

```

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [April 25, 2024, 11:42pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/7 "2024-04-25T23:42:34Z")

</div>

As suggested above, ask your sysadmin to allow access to [julialang.net](http://julialang.net) as well as [julialang.org](http://julialang.org).

---

<div class="post-metadata">

**Author:** ![Allan\_Baker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/allan_baker/32/42645_2.png) [@Allan\_Baker](https://discourse.julialang.org/u/Allan_Baker)\
**Post date:** [April 26, 2024, 12:16pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/8 "2024-04-26T12:16:16Z")

</div>

I get at 404 error. Even on my iPhone I can’t get to [julialang.net](http://julialang.net).

They attempted to go out to verify it but they can’t see the pages so they disallowed.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [April 27, 2024, 1:29pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/9 "2024-04-27T13:29:58Z")

</div>

Only the subdomain [storage.julialang.net](http://storage.julialang.net) is currently used so [julialang.net](http://julialang.net) is not assigned to anything, and yes that specific URL is a 404. This is not meant to be visited in your browser, it’s infrastructure. It is still a valid domain, just not a URL you can see. If you visit the [long julialang.net URL](https://storage.julialang.net/registry/23338594-aafe-5451-b93e-139f81909106/f8c090c98cada1c7542aa9c09c3e75e0e57a3121.tar.gz) that you posted above then it will initiate a download if you aren’t behind a firewall that blocks it.

Again: ask your sysadmin to allow access to at least the [storage.julialang.net](http://storage.julialang.net) domain and ideally all subdomains of [julialang.net](http://julialang.net) (in case we use more subdomains in the future). If they do this then the new setup will work, if they don’t then it won’t.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [April 27, 2024, 2:04pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/10 "2024-04-27T14:04:15Z")

</div>

Could we create a page on [julialang.org](http://julialang.org) or maybe [juliahub.com](http://juliahub.com) that states what domain names are officially associated with Julia? I’ve been fielding questions via email from system administrators and network operation managers for [contact@julialang.org](mailto:contact@julialang.org). The best I can do is point them here.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [April 30, 2024, 11:14am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/11 "2024-04-30T11:14:10Z")

</div>

> [@StefanKarpinski](#):
>
> Only the subdomain [storage.julialang.net](http://storage.julialang.net) is currently used so [julialang.net](http://julialang.net) is not assigned to anything, and yes that specific URL is a 404. This is not meant to be visited in your browser, it’s infrastructure. It is still a valid domain, just not a URL you can see.

I added an index page so that [https://storage.julialang.net](https://storage.julialang.net) is not a 404.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [April 30, 2024, 2:46pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/12 "2024-04-30T14:46:21Z")

</div>

> <https://github.com/JuliaLang/www.julialang.org/pull/2093>
>
> Also convert the IP address retention policy note into a section.

---

<div class="post-metadata">

**Author:** ![steph\_de\_paris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steph_de_paris/32/7780_2.png) [@steph\_de\_paris](https://discourse.julialang.org/u/steph_de_paris)\
**Post date:** [May 28, 2024, 8:08pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/13 "2024-05-28T20:08:10Z")

</div>

Hello, it is now impossible to install or update julia packages. Indeed, to install packages, we had to open gatekeepers to reach [github.com](http://github.com), [github.io](http://github.io), [julia-vscode.org](http://julia-vscode.org) and [julialang.org](http://julialang.org). Now it seems that we have to open new gatekeepers since we have now to reach new adress such as (?) [objects.githubusercontent.com](http://objects.githubusercontent.com). But it seems that this isn’t enough and that we need to open others, such as [storage.julialang.net](http://storage.julialang.net). Could you give me the exact list of all the sites we need to reach to update Julia Packages ? Note that if this list is dynamic, we won’t be able to manage it. Indeed, we’re under strict security constraints and can’t open gateways easily. Thank you in advance for the informations.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [May 28, 2024, 9:32pm UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/14 "2024-05-28T21:32:27Z")

</div>

See [Official Domains](https://julialang.org/downloads/#official_domains).

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [May 29, 2024, 2:44am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/15 "2024-05-29T02:44:13Z")

</div>

Access to GitHub domains is only needed if you want to install unregistered packages that are hosted on GitHub. You are not required to do that in any way. The fact that Pkg can get things from GitHub for you is a convenience. If GitHub now uses new domains like [objects.githubusercontent.com](http://objects.githubusercontent.com) that is not under our control and you can feel free to ignore it, just don’t expect direct installs from GitHub to work. If tomorrow they add some other subdomain, guess what? You’ll have to add that.

I don’t know about the [julia-vscode.org](http://julia-vscode.org) domain. Presumably that is for the Julia VS Code extension. If you don’t use that then you don’t need this. I also think it’s just a website? Not sure why it would be blocked in the first place.

That leaves exactly two domains that are needed for normal Pkg operations to work. There used to be one. Now, for technical reasons beyond our control, there are two. Sorry for the inconvenience. This was not done to annoy you. Believe me if there was a way to avoid having to have this conversation i would have preferred it.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 29, 2024, 3:45am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/16 "2024-05-29T03:45:02Z")

</div>

> [@StefanKarpinski](#):
>
> Access to GitHub domains is only needed if you want to install unregistered packages that are hosted on GitHub.

this might not be completely true if one is using JuliaUp, at least it tried to grab something from Github the other day when launched for the first time. (I hit this problem because I was on a weird system without `~/.git` access). It’s possible it might continue to work somehow, but just want to put this out here for the completeness of the discussion.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [May 29, 2024, 4:10am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/17 "2024-05-29T04:10:03Z")

</div>

> [@jling](#):
>
> this might not be completely true if one is using JuliaUp, at least it tried to grab something from Github the other day when launched for the first time.

Hmmm. We should try to identify those domains used by Juliaup, and then add them to the [Official Domains](https://julialang.org/downloads/#official_domains) section.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [May 29, 2024, 4:31am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/18 "2024-05-29T04:31:20Z")

</div>

Okay, so as far as I can tell, when you first install Juliaup, it downloads the Juliaup binaries from a URL of the form `https://github.com/JuliaLang/juliaup/releases/latest/download/*`.\[1\]

Therefore, the first time that a user installs Juliaup, they need to be able to make network requests to the `github.com` domain.

So, we should probably add a note (to the [“Official Domains” section](https://julialang.org/downloads/#official_domains) that informs users that they need to be able to make network requests to the `github.com` domain, but only if they are using Juliaup.

If the user is not using Juliaup, then they do not need access to the `github.com` domain.

* * *

I’m not sure if Juliaup also makes requests to `github.com` when updating Juliaup itself, or if access to `github.com` is only needed when installing Juliaup for the first time. @davidanthoff Do you know the answer to this question?

* * *

1. See [line 84 of this file](https://github.com/JuliaLang/juliaup/blob/13fa656f43c2feb9d85882def899eeda968537b9/deploy/shellscript/juliaup-init.cmd#L84) for the place where this URL is defined.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [May 29, 2024, 4:45am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/19 "2024-05-29T04:45:33Z")

</div>

For Juliaup, as far as I can tell, there are no other domains needed by Juliaup, except for `github.com` (as mentioned in my previous comment), and of course `*.julialang.org` and `*.julialang.net`.

If someone finds an additional domain (or subdomain) that is needed for Juliaup to work, please let me know.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [May 29, 2024, 11:22am UTC](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378/20 "2024-05-29T11:22:30Z")

</div>

Would probably make sense to serve juliaup from s3 just like the regular binaries. Then those would also be a [julialang.org](http://julialang.org) subdomain.

[Next page](https://discourse.julialang.org/t/changes-to-pkgservers-storage-backend/112378.md?page=2)