Pkg fetching takes very long at 99.9% @ v1.6 (Win 10)

[essenciary]

Should this be reported as a bug or is it to be expected with RC?

[helgee]

It seems that you are still using the old (slow) Git-based registry download. Try this maybe?

Pkg.update in Julia 1.5 uses github

You probably have the General registry as a git repo. Delete it and install it again.

[carstenbauer]

Unfortunately, I have to confirm that it sometimes takes forever to download the registry on Windows. I’ve observed this even on a fresh machine with no previous local (git) registry and also when setting ENV["JULIA_PKG_SERVER] = "" and so on. Internet connection was fine and can’t explain the behaviour. This really spoils the Julia experience right from the start.

Oddly, I believe to have observed (not sure) that the title of the Julia REPL window (i.e. powershell window) sometimes say something like “Select user” or similar. Randomly hitting “Enter” has in some cases continued/sped up the download. But maybe I’m hallucinating… Would be great if someone could investigate this systematically. (Too speculative)

[carstenbauer]

Just to contribute more substantial evidence to the discussion:

I tried to make the gif match “real time”. It hangs at 99,9% for about 2 minutes. This is Julia 1.6 on windows 10 with no prior Julia installation (i.e. no ~/.julia directory).

(BTW, I don’t think it is 1.6 specific. I also saw this effect on 1.5 already.)

[helgee]

I nuked my registries (on 1.5 and 1.6) and tried again. Same results here. The initial installation takes ages.

[kristoffer.carlsson]

Check if Windows Defender is taking a lot of CPU.

[carstenbauer]

Indeed, turning off Windows Defender reduces the time quite noticably to about 30 seconds. I would still say that it could be faster (it is on macOS) but it is probably acceptable like this. So I guess the question is how to avoid the chiming in of Windows Defender.

(BTW, Windows Defender is not using terribly much CPU. For me it was around 30 %)

[essenciary]

I have ~/.julia/ added as an exception in Windows Defender.

[kristoffer.carlsson]

What will probably happen is that files will be read straight out of the tar ball downloaded (see Tar.extract_file: extract content of single file in tarball by fredrikekre · Pull Request #95 · JuliaIO/Tar.jl · GitHub) which would avoid having to materialize so many files (which is what causes Windows Defender to be unhappy). Might take a while for that though.

[sdewaele]

Thanks for the suggestion! Just to verify - is this the same as “Add an exclusion to Windows Security” described here?

[deszoeke]

I have the same problem with Julia 1.7.2 on linux. It eventually works after spending about 2 min at Fetching…99.9%

[suavesito]

That’s… weird. Can you elaborate more into if you were running a lot of other processes?, or maybe a big IO operation? HDD vs SSD?

[StefanKarpinski]

Try deleting your registries and re-adding them, which will switch to using a tarball instead of a git clone.

[deszoeke]

Yes, thanks. Deleting the registries as in

Pkg.update in Julia 1.5 uses github

You probably have the General registry as a git repo. Delete it and install it again.

speeds it up so the time is nearly unnoticeable.

I would not have expected this slow git registry download to affect a new install of Julia 1.7.2.

[StefanKarpinski]

That only happens if you have a git cloned registry, which either means you manually created a git cloned registry at some point, in which case we figure you know what you’re doing, or that you have been using that registry since many Julia versions ago without deleting your registries, which is possible but seems somewhat unlikely.

[avik]

This is quite an old thread, but since people are talking about windows defender, it’s probably worthwhile to mention here that Pkg downloads and extracts the registry tar bundle in a temporary directory. This means that excluding .julia from Windows Defender is not sufficient. The tar bundle contains tens of thousands of small files, and that is what causes Windows Defender to lock up for minutes.

Newer versions of Julia operate by not extracting the files in the tar bundle, which should be helpful.

[jdad]

Only for Julia >= 1.7, cf.

• Registries downloaded from the Pkg Server (not git) are no longer uncompressed into files but instead read directly from the compressed tarball into memory. This improves performance on filesystems which do not handle a large number of files well. To turn this feature off, set the environment variable JULIA_PKG_UNPACK_REGISTRY=true .

In https://github.com/JuliaLang/julia/blob/v1.7.0/NEWS.md