Very slow "Updating registry at 'C:\Users\me\.julia\registries\General'"

devel-chm · December 10, 2020, 3:18pm

I’m using win10 and julia-1.5.3 from the REPL. When I use Pkg mode to add a new package (or other operations) it starts by updating the registries\General:

(@v1.5) pkg> add Pluto
Updating registry at C:\Users\me\.julia\registries\General

where the updating line can take multiple minutes…every…time…

This seems to be a regression from Pkg adds from a year ago.

Is there a debug option to see what is taking so long?
Has anyone else seen this slow down?
Is there a fix/solution to the problem?

NOTE: This can be very problematic when I load a Pluto notebook and it starts by adding the needed packages to the session!

kristoffer.carlsson · December 10, 2020, 3:59pm

You can do the following:

pkg> registry rm General

julia> ENV["JULIA_PKG_SERVER"] = ""

pkg> registry add General

Now it will update using Git which should be faster on windows. Only need to do this once.

devel-chm · December 10, 2020, 4:38pm

Looks promising so far! The full clone of General.git process took less time than the update via the package server. I assume git updates will be even faster going forward. Thanks!

Heinz_Graf · February 15, 2021, 9:57am

Hello Kristoffer
I have a similar Problem. I use also win10 and Julia-1.5.3. It worked perfect during years. Since about 10 days, I can’t load Packages from the General registry e.g. Plots and others (see next)

(v1.5) pkg> add Plots
Updating registry at d:\Users\Heinz\.julia\registries\JuliaComputingRegistry
ERROR: The following package names could not be resolved:

Plots (not found in project, manifest or registry)

(v1.5) pkg> st --project
Status D:\Users\Heinz\.julia\environments\v1.5\Project.toml
[44cfe95a] Pkg
[9a3f8284] Random

(v1.5) pkg> st --manifest
Status D:\Users\Heinz\.julia\environments\v1.5\Manifest.toml
[2a0f44e3] Base64
[ade2ca70] Dates
[b77e0a4c] InteractiveUtils
[76f85450] LibGit2
[8f399da3] Libdl
[56ddb016] Logging
[d6f4376e] Markdown
[44cfe95a] Pkg
[de0858da] Printf
[3fa0cd96] REPL
[9a3f8284] Random
[ea8e919c] SHA
[9e88b42a] Serialization
[6462fe0b] Sockets
[cf7118a7] UUIDs
[4ec0a83e] Unicode

(v1.5) pkg>

(v1.5) pkg> registry st
Registry Status
[bbcd6645] JuliaComputingRegistry (https://github.com/JuliaComputing/JuliaComputingRegistry)

As you suggested, I removede the registry and then added it again. Then

(v1.5) pkg> registry add General
Cloning registry from “https://github.com/JuliaRegistries/General.git”
ERROR: failed to clone from https://github.com/JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:OS, failed to send request: Der Servername oder die Serveradresse konnte nicht verarbeitet werden.

Pleas can you help me?
Thanks!

kristoffer.carlsson · February 15, 2021, 6:33pm

Are you using JuliaPro or standard Julia?

Heinz_Graf · February 15, 2021, 7:51pm

Hi,
I’m using standard Juli. After having had this problem, I tried also Julia Pro.But there I had the same kind of errors.
I checkt “registry”, but with the following error messages, I coud’t get further.

(v1.5) pkg> registry up
Updating registry at d:\Users\Heinz\.julia\registries\JuliaComputingRegistry

(v1.5) pkg> add Plots
ERROR: The following package names could not be resolved:

Plots (not found in project, manifest or registry)

kristoffer.carlsson · February 15, 2021, 8:53pm

Try clear out d:\Users\Heinz\.julia\registries completely.

Other than that, are you behind some proxy or got any special network settings that might interfere with the registry download?

Heinz_Graf · February 15, 2021, 9:40pm

Thanks a lot, I will try this tomorrow.
To your question concerning special network settings: Can the LOAD_PATH setting have some influence on the observed behavior and how must this setting be? How can the LOAD_PATH be changed?

erans · February 16, 2021, 12:09am

This does indeed seem to work. I’m curious though, prior to registry rm General this was my registry:

pkg> registry st
Registry Status
 [23338594] General (https://github.com/JuliaRegistries/General.git)

is this not already Git?

In fact, this is the exact same output I get after making the change. Why is the update time all of a sudden so different?

Heinz_Graf · February 16, 2021, 9:00am

Good morning Kristoffer
Thank you again for your hint. It worked!
After deleting “registry” and deinstalling julia 1.5.3, I reinstalled Julia 1.5.3.
After starting Julia and “registry status” there was no registry. Then I added registry with “registry add General” and it was done!
The reason for the problems are still unknown. But lets now move on!
Thank you again for your help.

PS. A good side-effect was, that I learned a lot more about Pkg.jl and the Registry!

Heinz_Graf · February 16, 2021, 9:03am

Good morning Kristoffer
Thank you again for your hint. It worked!
After deleting “registry” and deinstalling julia 1.5.3, I reinstalled Julia 1.5.3.
After starting Julia and “registry status” there was no registry. Then I added registry with “registry add General” and it was done!
The reason for the problems are still unknown. But lets now move on!
Thank you again for your help.

PS. A good side-effect was, that I learned a lot more about Pkg.jl and the Registry!

kristoffer.carlsson · February 16, 2021, 10:42am

It should be, I am not sure why it didn’t work.

erans · February 16, 2021, 7:42pm

Hmm…

As a side note, when I added a package to my default environment today the registry update again took well over a minute

I suppose this is not the end of the world, but a bit frustrating that simply communicating with the registry comes with so much time overhead.

DNF · February 16, 2021, 8:00pm

Perhaps this is off-topic, but I found the Windows experience frustrating enough that I installed Julia to WSL instead. So far this is a significant improvement over ‘ordinary Windows’, shell mode is nicer, too.

I wonder if there are any drawbacks, or is this basically the best solution for windows?

kristoffer.carlsson · February 17, 2021, 10:20am

It’s bad on Windows. Should be better in the upcoming 1.6. For now you can do:

julia> ENV["JULIA_PKG_SERVER"] = ""

pkg> registry rm General

pkg> registry add General

and you will get a git version of the registry instead which should be faster to update.

magister-ludi · February 17, 2021, 12:06pm

Depending on what is required, WSL is probably a good solution. I did try it and the improvements in speed were very good.

However I also run a Linux installation in VirtualBox, and it turns out that VirtualBox and WSL can’t be installed at the same time. I had to choose between the two, so I chose VirtualBox.

Both Linux’y systems were much faster than Windows (all running on the same hardware, of course). Without any real knowledge on my part, my guess is that a lot of this is because the Windows file system is really slow.

erans · February 22, 2021, 11:06am

This is what I already did…

erans · February 22, 2021, 11:07am

Thanks for the tip, but unfortunately I’m doing some of this setup for work and trying to convince our IT that we should have WSL installed is just not a battle I’m willing to wage at the moment

erans · April 10, 2021, 9:57pm

Hi @kristoffer.carlsson,
just as a follow up to the comment that things should be better in 1.6.
I’m still experiencing registry update times in the minutes range (1-2 I’d say).

Not the end of the world since this is not something that’s done constantly, but it does seem I bit slower than I would have expected.