Why is the update of the registries so slow?

I did this but still the fetching takes about 1 min.

On a slowish laptop, with Wi-Fi connection, Windows 10: Full registry after a removal of the entire .julia folder takes 2 1/2 minutes.

Try with Windows Defender disabled.

1 Like
4 Likes

Iā€™m doing package updates on Debian Linux using the default Debian packages (julia 1.5.3) behind a proxy and itā€™s saying

   Updating registry at `~/.julia/registries/General` 

For about 10 minutes so far. This is on a gigabit fiber connection.
Then all of a sudden it says:

   Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Fetching: [========================================>]  100.0 %

that bar flew across. clearly itā€™s just hanging for some dumb reason like a timeout not because of anything real.

Now itā€™s hanging some more.

Anyone with behavior like this?

(note: My home directory is on an NFS4 server running on a glusterfs cluster)

2 Likes

First of all, do not use the default debian packages but the latest stable release from julialang.org .
Easy to install:

bash -ci "$(curl -fsSL https://raw.githubusercontent.com/abelsiqueira/jill/master/jill.sh)"

Secondly, check the server status:
https://status.julialang.org

You could try a different server.

But if you are behind a proxy AND on an nfs server, both of it could cause problems where I cannot help you.

1 Like

Iā€™ll start with the latest julialang version and see where that gets me.

btw https://status.julialang.org times out, ā€œtaking too long to respondā€ so maybe thatā€™s the issue.

Perhaps julialang.org is blocked by your proxy. Ask your admin. Or just use your phone to get internetā€¦

I am the admin, and no itā€™s not blocked. It came through eventually, but apparently itā€™s intermittent for me.

This bar graph seems to indicate some issues recently:

Perhaps for some reason Iā€™m getting connected to eu-central instead of us-west?

Oh us-west also shows issues:

Alternatively set it so a different Pkg Server, e.g.

export JULIA_PKG_SERVER=us-east.pkg.julialang.org

in the shell, or

ENV["JULIA_PKG_SERVER"] = "us-east.pkg.julialang.org"

from within Julia.

Ok, if all of them have problems you can only complain here in the forum and wait until the problems are fixedā€¦

3 Likes

The time is the same here, although I am using JuliaPro_v1.5.4-1 on Windows and was adding a package. The first thing Julia did was to spend 11 min updating General. Then it took only a few seconds updating JuliaComputingRegistry and the rest of stuff. This was not the first time it took about 10 min updating General.

Might be a Windows Defender thing: Pkg fetching takes very long at 99.9% @ v1.6 (Win 10) - #7 by carstenbauer

Iā€™ve been seeing the same thing on Linux over the last days though (although as usual worse on Windows).

If itā€™s happening on Linux as well, it sounds like a network/server issue, not the Windows problem that unpacking a tarball is ungodly slow. Fortunately, the tarball unpacking issue should be a thing of the past on all platforms with the 1.7 release since weā€™re now leaving the registry tarballs as-is on disk instead of unpacking them: https://github.com/JuliaLang/Pkg.jl/pull/2431.

6 Likes

Just tried again on 1.6.1 on a Manjaro machine, again got stuck - this time on 100% rather than 99.9% - for about 4 minutes. My internet is a bit slow this morning but still clocks in at about 15MB/s, which I would have thought should be well enough?

How fast should I expect an update of the general registry to be assuming everything works normally (and Iā€™m not on Windows)?

Would you be able to try the 1.7 beta or compile the release-1.6 branch form source? There are some fixes to Downloads that should be included in both of those and it would be interesting to see if that improves things for you.

1.7beta seemed a bit faster, although still got stuck for a noticeable time on 100% but a marked improvement!

The problem I was having was that the us-west package server had a problem with itā€™s IPv6 connectivity. Perhaps others are having similar issues on other PKG servers?

See this thread [resolved] Problem with the IPv6 connectivity of the us-west Pkg server

Hi, recurrent issue indeed. Often stuck on a linux computation server with > Go connection. Painful.

1 Like

Iā€™m having the same problem on Windows. Is there a list of package servers? I live in southern Europe. Iā€™d like to try changing the server before trying anything else. (I have an almost 200mbs working conncection and Iā€™m on the latest available version of Julia: 1.6.4)