Why is the update of the registries so slow?

General Usage
1 
$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.2.0 (2019-08-20)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> t = time(); include("go.jl"); println("$(time()-t)")
Activating environment at `C:\Users\PK\Documents\work\ACol\Project.toml`
  Updating registry at `C:\Users\PK\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
45.980000019073486

Fourty six seconds. Why?

1 Like
2

Slow internet connection? Mine usually completes in ~5 sec

3

This is on 10 Mpbs. You?

1 Like
4

There is definitely a minute long lag with the package manager every now and then for me. This is especially true if I use the package manager after first loading or precompiling a larger package. That’s why it seems to me that sometimes it is better to open a new instance of Julia to get a quicker response from Pkg.

Also, I’m on Linux not Windows.

5 Likes
5

Looks like you are using Windows: LibGit2 operations are generally pretty slow there from my experience.

1 Like
6

Is it just a Windows thing? I had trouble explaining to my students why installing packages in a fresh Julia install takes over 1 minute on the Windows school computers, but only a few seconds on my Linux laptop. That would explain it.

2 Likes
7

Yes, git is (of course) very snappy on Linux

8

Could people on Windows look at the process monitor while the registry is updating and see if Windows Defender is taking up a lot of CPU during that.

Also, does adding the .julia/registries/General directory to the exclusion list of Windows Defender help at all?

2 Likes
9

It’s not only a windows issue, I’d say:

image
image954Ă—501 25.1 KB

The article:

Hard to compare package installations between machines, but after Pkg3, I’ve never noticed windows to be particularly slow anymore…

1 Like
10

Can you share the toml files for this environment (before the first up)?

11

Good thinking. During the fetching of the registry the “anti-malware service executable” was over 30% of CPU usage.

I added the folder .julia/registries to the exclusion list. That didn’t work.
I added .julia to the exclusion list, and then the fetching of the registries did not result in high anti-malware executable CPU usage. However when the fetching was done, the anti-malware executable went up to over 50% of CPU usage.

12

I have no explicit comment to your question. But I would say Defender, Anitmalware blabla (possibly this is the same thing), and some other windows processes that I do not understand (wmi host provider, etc.) tend to slow Julia (and other software) by a huge margin.
I actually have the feeling, Microsoft has a deal with energy providers.

13

Turning off antivirus software (not necessarily advisable), resulted in a speedup of around 3 (16 seconds compared to 46 seconds originally). I suspect there is an additional speedup that might be gained by turning off indexing (the Windows indexer is running towards the end of the instantiation).

3 Likes
14

Long time ago the registry (METADATA) was a wide (~2500 subdirectories) git repository with small individual files; so a database as directory structure. This took very long to update via git pull.

The Pkg3 happened. And everything went to be way-more-efficient.

Now the Registry is a tree (1-level Intial char, 2-nd level Package name) of small individual files, so a database as directory structure. Could it be that this takes long as updates via git pull?

15

Will this get faster? I still find it is much less responsive than python and r on windows.

16

Yes. You can already use a Julia 1.4 prerelease and set JULIA_PKG_SERVER to geo.pkg.julialang.org to get registries as tarballs.

3 Likes
17

The link, when clicking on it, redirects me to eu.pkg.julialang.org which doesn’t seem to exist.

1 Like
18

It’s not a website, it serves packages.

5 Likes
19

I am using Julia 1.4.0 but the fetching process of packages updating is still very slow.

20

If you set $ export JULIA_PKG_SERVER=pkg.julialang.org (and delete the old git-registry) it should be significantly faster.

Edit: I see this was discussed above, but maybe the problem is that your registry is still in git-form?

1 Like