Why is the update of the registries so slow?

Is it just a Windows thing? I had trouble explaining to my students why installing packages in a fresh Julia install takes over 1 minute on the Windows school computers, but only a few seconds on my Linux laptop. That would explain it.

2 Likes

Yes, git is (of course) very snappy on Linux

Could people on Windows look at the process monitor while the registry is updating and see if Windows Defender is taking up a lot of CPU during that.

Also, does adding the .julia/registries/General directory to the exclusion list of Windows Defender help at all?

2 Likes

It’s not only a windows issue, I’d say:


The article:

Hard to compare package installations between machines, but after Pkg3, I’ve never noticed windows to be particularly slow anymore…

1 Like

Can you share the toml files for this environment (before the first up)?

Good thinking. During the fetching of the registry the “anti-malware service executable” was over 30% of CPU usage.

I added the folder .julia/registries to the exclusion list. That didn’t work.
I added .julia to the exclusion list, and then the fetching of the registries did not result in high anti-malware executable CPU usage. However when the fetching was done, the anti-malware executable went up to over 50% of CPU usage.

I have no explicit comment to your question. But I would say Defender, Anitmalware blabla (possibly this is the same thing), and some other windows processes that I do not understand (wmi host provider, etc.) tend to slow Julia (and other software) by a huge margin.
I actually have the feeling, Microsoft has a deal with energy providers.

Turning off antivirus software (not necessarily advisable), resulted in a speedup of around 3 (16 seconds compared to 46 seconds originally). I suspect there is an additional speedup that might be gained by turning off indexing (the Windows indexer is running towards the end of the instantiation).

3 Likes

Long time ago the registry (METADATA) was a wide (~2500 subdirectories) git repository with small individual files; so a database as directory structure. This took very long to update via git pull.

The Pkg3 happened. And everything went to be way-more-efficient.

Now the Registry is a tree (1-level Intial char, 2-nd level Package name) of small individual files, so a database as directory structure. Could it be that this takes long as updates via git pull?

Will this get faster? I still find it is much less responsive than python and r on windows.

Yes. You can already use a Julia 1.4 prerelease and set JULIA_PKG_SERVER to geo.pkg.julialang.org to get registries as tarballs.

2 Likes

The link, when clicking on it, redirects me to eu.pkg.julialang.org which doesn’t seem to exist.

1 Like

It’s not a website, it serves packages.

5 Likes

I am using Julia 1.4.0 but the fetching process of packages updating is still very slow.

If you set $ export JULIA_PKG_SERVER=pkg.julialang.org (and delete the old git-registry) it should be significantly faster.

Edit: I see this was discussed above, but maybe the problem is that your registry is still in git-form?

I did this but still the fetching takes about 1 min.

On a slowish laptop, with Wi-Fi connection, Windows 10: Full registry after a removal of the entire .julia folder takes 2 1/2 minutes.

Try with Windows Defender disabled.

1 Like
4 Likes

I’m doing package updates on Debian Linux using the default Debian packages (julia 1.5.3) behind a proxy and it’s saying

   Updating registry at `~/.julia/registries/General` 

For about 10 minutes so far. This is on a gigabit fiber connection.
Then all of a sudden it says:

   Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Fetching: [========================================>]  100.0 %

that bar flew across. clearly it’s just hanging for some dumb reason like a timeout not because of anything real.

Now it’s hanging some more.

Anyone with behavior like this?

(note: My home directory is on an NFS4 server running on a glusterfs cluster)

2 Likes