One of the improvements I was most looking forward to in v1.5 is the new Pkg infrastructure with expected speed improvements for non-US users. But so far my end-user experience is largely unchanged. Is this expected, or should things be significantly faster by now? Maybe the new servers haven’t come online yet?
For example, in the (amazing) Juliacon Pluto video, the response time for running ]add Pluto was instantaneous. I almost spurted coffee all over my screen when I saw it. But this is what I am seeing here (on a 100 Mbps fiber line in Sweden which has very good internet infrastructure in general):
Are you on windows? There are some known issues, mainly due to file system performance. If so, it would be interesting to see the speed of Julia on wsl.
Completely remove .julia. Then,
for one particular package (Elfel.jl), running activate and instantiate, I get these measurements:
Windows 10 126 s
WSL 1 131 s
WSL 2 13 s
I don’t think it has changed a lot in the EU, it has always been fast for me, and still is. For example, here is an add of Pluto, with downloading all it’s dependencies and updating the registry:
It looks like you still have the registry as a git repo, so registry updates does not use the Pkg servers. In my experience, libgit2 is very slow on Windows, so that is probably why it takes time for you. Is it still slow the second Pkg operation which doesn’t do a registry update?
If you delete the registry and install it again you should get a Pkg server registry, but then you might run into problems with WIndows virus scanning (Windows Defender?) that don’t like that we unpack a tarball with many files. That might be a reason for slow package downloads too, if the packages contain many files.
It would be great if someone could dig into what’s actually causing the slowness here. Is it the download or the unpacking? Is it only when the virus scanner is running or always? Does it actually depend on geographic location?
I measured fresh install on Windows 10, WSL 1 and 2.
All of these measurements with the antivirus software running.
As you can see, WSL 2 is an order of magnitude faster. That would seem to indicate
access to disk as the limiting factor. According to the info I found on the web, WSL 2 uses “native Linux filesystems”.
On Windows 10, a fresh registry install in the central U.S. takes 158 seconds for me on latest nightly using the new package server. Looking at Task Manager, the antivirus program seems to be upset when we’re unpacking all the tiny .toml files (something like 16k files & 4k folders totaling 5 MB on disk).
There’s also a lengthy delay (similar to the ~3 minute delay for registry installation) that seems to be occasionally triggered when performing the first package operation within a given Julia session, even if the registry was just updated in another session. I haven’t managed to isolate the trigger, so it’s not reproducible, but the antivirus is similarly active for the duration of the delay.
I added the file type *.toml to the exclusions of the Microsoft Defender Antivirus. The Windows 10 registry install time dropped from 126 seconds to 57 seconds. The WSL 2 time is still at 13 seconds.
Thanks everyone for all the suggestions and tests. Below are some of my own. I can also confirm that Windows Defender/Real-time Protection (“RTP”) makes things really slow, but my timings were still pretty bad when RTP was completely disabled. Note that RTP can only be disabled temporarily, it reenables itself after a few minutes.
Setup: Windows 10 laptop with Julia 1.5.1, located in Sweden, no antivirus other than Windows Real-time Protection.
Procedure:
Delete ~/.julia
Start Julia, then immediately…
First timing: ]add Pluto (i.e. including initial registry setup).
Second timing: ]add Revise (immediately after Pluto)
Restart Julia. (Seems to make the next step update the registry - I think?)
Third timing: ]add Literate
Every run (i.e. the entire sequence of steps 1-6) was repeated a few times to get a range of timings.
Windows 10, RTP on: 170-180s + 2-3s + 6-184s
Windows 10, RTP off: 45-55s + 2-3s + 4-5s
wsl1, RTP on: 190-210s + 2-8s + 10-14s
wsl1, RTP off: 70-80s + 2-5s + 3-9s
wsl2, RTP on: 13-15s + 2-3s + 5-6s
wsl2, RTP off: 12-13s + 2-3s + 4-5s
For the longer wsl1 and vanilla Windows10 tests, nearly all the time was spent “Installing known registries”. During this time I kept an eye on the contents of registries/General: it populated evenly with single-letter folders during the entire time. Not sure if this means it was downloading or unpacking as @StefanKarpinski was wondering, but I assume the latter. The same thing happened on the occasions when “Updating registry” took a long time.
EDIT: By the way, my sympathies to the Pkg team. It must be rough doing tons of clever stuff to speed things up only to have Windows crap all over it - and then having people like me who are stuck on Windows wonder why it isn’t faster…
Hmm I was going to run my own tests but it seems the servers are actually down currently:
(@v1.4) pkg> up
Updating registry at `C:\Users\Jeremy\.julia\registries\General`
┌ Warning: could not download https://pkg.julialang.org/registries
└ @ Pkg.Types D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Types.jl:889
This is the unpacking. The download is just a normal download of tar.gz file.
Thanks for saying so. This is unfortunate. Based on this feedback it seems like the old way of updating registries via git was probably faster because git updates everything in place and only touches files that have actually changed—which is typically not that many. The new process downloads and extracts a new tarball each time, which is actually significantly faster on Linux and macOS than doing the whole git update negotiation. But not on Windows… which is a bit ironic because git is notoriously horribly slow on Windows. But apparently faster than what we’re now doing.
So mitigation options: for those using 1.5 on Windows, if you clone ~/.julia/registries/General as a git repo then it will update it as a git repo which should be as fast as it was before. We could change 1.5.2 to always do this on Windows since this is so horribly slow. Issue: Pkg.jl#2014.
In 1.6 I’d like to stop extracting the tarball altogether and just read the registry directly form the packed, compressed tar.gz file. That’s likely to be faster on all platforms, but especially on Windows.
My later measurements make me revise my earlier statement: The slowdown on Windows seems to be due to two causes.
Real-time protection (antivirus, RTP).
Disc access.
WSL 2 measurements are hardly affected by turning off RTP. It would appear that those files are written into Linux own file system, and are not visible to RTP.
WSL 1 and Windows itself has the manipulation of the registry files seriously hampered by RTP, because the files are written into the Windows file system.
Turning off RTP does not make up all the difference relative to WSL 2. It is still almost 3 times faster than Windows itself. That might be due to the difference in reading/writing thousands of files into different types of file systems.
I just found this, which appears to be an exact match of our problem. Note that they deployed a fix 3 weeks ago that just involves continuously updated Defender signatures.
