[resolved] Problem with the IPv6 connectivity of the `us-west` Pkg server

dlakelan · June 19, 2021, 8:33pm

I’m having problems updating packages, status.julialang.org takes a long time to respond, sometimes doesn’t respond, it does say that things are operational, but the detailed graphs show lots of red bars for a while now.

Further useful info perhaps:

Not able to ping us-west.pkg.julialang.org:

dlakelan@tintin:~$ ping us-west.pkg.julialang.org
PING us-west.pkg.julialang.org(2600:1f13:a21:3e00:e267:cc2f:ada8:f580 (2600:1f13:a21:3e00:e267:cc2f:ada8:f580)) 56 data bytes
^C
--- us-west.pkg.julialang.org ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5098ms

whereas pinging say google.com works fine so it’s not like I have no internet, or no working ipv6

dlakelan@tintin:~$ ping us-west.pkg.julialang.org
PING us-west.pkg.julialang.org(2600:1f13:a21:3e00:e267:cc2f:ada8:f580 (2600:1f13:a21:3e00:e267:cc2f:ada8:f580)) 56 data bytes
^C
--- us-west.pkg.julialang.org ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5098ms

ping4 of hte package server doesn’t work either:

dlakelan@tintin:~$ ping4 us-west.pkg.julialang.org
PING  (52.24.172.208) 56(84) bytes of data.
^C
---  ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3080ms

(of course, maybe you’ve just configged it not to respond to pings?)

The main symptom from Julia is that ]update leads to:

(@v1.6) pkg> update
Updating registry at ~/.julia/registries/General

which hangs for tens of minutes.

logankilpatrick · June 19, 2021, 9:39pm

Cc @staticfloat @fredrikekre @dilumaluthge (I think some of you folks are involved in this)

dlakelan · June 19, 2021, 9:41pm

I deleted .julia/registries/General and did a ]update it’s now been doing:

  Installing known registries into `~/.julia`

for quite a while, we’ll see, this might be something specific to me or my system, but the fact that those history graphs show so much red and that status.julialang.org doesn’t respond some of the time seems suspect

dilumaluthge · June 19, 2021, 9:46pm

Yeah it is strange that the status page says “operational” when there are so many red bars.

I’ve rebooted most of the Pkg servers. I’ve also rebooted the machine that status.julialang.org runs on.

dilumaluthge · June 19, 2021, 9:50pm

Instead of pinging the Pkg servers, you can try navigating to them in a web browser.

For example, I just tried going to https://us-west.pkg.julialang.org in a web browser, and I received the following message:

You have reached a Julia package server. It is not meant for browsing but for serving resources required by Julia clients to install packages. To discover and explore the Julia package ecosystem, see this page for a list of services.

This indicates that my machine is able to successfully reach the us-west Pkg server.

dilumaluthge · June 19, 2021, 10:17pm

Anyway, you could try getting the registry from Git instead of from the Pkg server:

julia> ENV["JULIA_PKG_SERVER"] = ""

julia> import Pkg

julia> Pkg.Registry.rm("General")

julia> Pkg.Registry.add("General")

julia> Pkg.Registry.update()

dlakelan · June 19, 2021, 10:30pm

Yeah, I’m doing something like that now. I believe that it’s taking forever and a day.

When I hit us-west.pkg.julialang.org in my browser, it has connection timed out… retrying it eventually responded after about 40 seconds. But then a second try, and it timed out again.

dilumaluthge · June 19, 2021, 10:35pm

Just double-checking: you’re not behind any kind of firewall or anything like that, right?

dlakelan · June 19, 2021, 10:37pm

Nothing that is filtering us-west.pkg.julialang.org for example. It did just load finally. Basically it seems intermittent. It responds maybe one out of 4 or 5 times I try to connect and when it does respond it takes tens of seconds to do so.

All my normal web browsing and access to everything I do normally (such as discourse.julialang.org) is working fine.

dilumaluthge · June 19, 2021, 10:41pm

But you are behind some kind of firewall, right?

Can you try from a different machine that is not behind any kind of firewall whatsoever?

dilumaluthge · June 19, 2021, 10:41pm

I’m not able to reproduce the results that you are seeing. https://us-west.pkg.julialang.org/ loads for me every time, and within a few seconds.

We probably need more data. To other people reading this thread: can you try going to https://us-west.pkg.julialang.org in your web browser and seeing if it loads, and if so, how long it takes to load.

dlakelan · June 19, 2021, 10:52pm

I have a router, so yes. Every sane person is behind some kind of firewall. No one just plugs their laptop directly to their cable modem or whatever. The best I can do is try to grab it from my phone with wifi turned off, but that will not be primarily testing the firewall it’ll be testing the difference between ATT connectivity and T-Mobile connectivity.

from my phone on t-mobile it replies within 3 to 5 seconds.

It’s very possible that there’s an issue with ATT connectivity to the servers. However, notice the output from mtr:

Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                     Packets               Pings
 Host                                                                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2600:1700:4810:877f::1                                                                                          0.0%   156    0.4   0.2   0.2   0.4   0.0
 2. (waiting for reply)
 3. 2001:506:6000:131:71:156:211:58                                                                                 0.0%   155    2.8  10.1   2.8 260.1  26.1
 4. 2001:506:6000:131:69:235:127:154                                                                               18.2%   155  9929. 7115. 3028. 9938. 2381.
 5. 2001:1890:ff:ff06:12:242:115:36                                                                                41.3%   155    4.1   4.4   4.1   7.4   0.5
 6. 2001:1890:c02:3d0d::117c:aa09                                                                                   0.0%   155    5.4   7.2   5.3  44.0   4.6
 7. (waiting for reply)

Shows that the last hop before the server gives 0% packet loss and avg 7.2 ms ± 4.6ms of ping, so I don’t think there’s a route issue. (the intermediate nodes showing packet loss is just due to people down-prioritizing pings on core routers etc)

Also nothing about my firewall would be causing those red bars on status.julialang.org

Also I was finally able to accomplish a pkg update, it took about an hour.

dilumaluthge · June 19, 2021, 10:56pm

Yeah I think there are two different questions here:

Why is status.julialang.org showing so many red bars? In particular, why is the status page showing red bars for servers that I am able to reach from my machine.
Why is your machine having trouble connecting to the us-west package server?

dlakelan · June 19, 2021, 10:58pm

Exactly. All I can tell you is that to the extent that there’s a problem connecting, it’s 100% to the package servers, everything else I try to connect to is totally fine.

Examples of things that work fine for me:
http://discourse.julialang.org

signal messaging
checking email
Slack boards
craigslist
statmodeling.stat.columbia.edu
etc.etc.etc

dilumaluthge · June 19, 2021, 11:08pm

Interesting.

The us-west.pkg.julialang.org machine is hosted by AWS. Specifically, it is located in one of the AWS regions in the western United States.

But looking at the AWS status page (https://status.aws.amazon.com/), I’m not seeing any relevant outages.

contradict · June 19, 2021, 11:08pm

(Responding from your request on Slack)

Loads quickly from Los Angeles:

$ time wget -q https://us-west.pkg.julialang.org/ -O /dev/null

real    0m0.143s
user    0m0.001s
sys     0m0.009s

On several tries the slowest was 0.152s.

dlakelan · June 19, 2021, 11:37pm

same thing from my house, also in the LA area, but via ATT fiber (@contradict who’s your ISP? and do you have Ipv6?)

dlakelan@tintin:~$ time wget -q https://us-west.pkg.julialang.org -O /dev/null

real	0m32.982s
user	0m0.013s
sys	0m0.008s

brenhinkeller · June 20, 2021, 1:59am

From northeast US:

$ time wget -q https://us-west.pkg.julialang.org/ -O /dev/null

real	0m0.346s
user	0m0.026s
sys	0m0.012s

dilumaluthge · June 20, 2021, 2:01am

Oh, interesting… maybe this has something to do with IPv4 versus IPv6.

I do not have IPv6. Does anyone else in this thread have IPv6?

jpsamaroo · June 20, 2021, 2:47am

Testing with curl, IPv4 works fine, but IPv6 just hangs. I’m on Google Fiber, and a public IPv6 test website shows my IPv6 as functioning normally.