Fetching progress: why does it almost always stall at 99.9%?

General Usage
1

When I install Julia packages, sometimes fetching occurs. And when fetching occurs, a lot of times it progresses quite fast until 99.9 % (within a few seconds), but then it stalls there for several minutes:

(@v1.8) pkg> add CairoMakie
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Fetching: [========================================>]  99.9 %

Why is this so? If there are really so many things to take care of around the 99.9 % progress mark, shouldn’t the progress bar have progressed more slowly in the earlier portion of the fetching process? Can we make the fetching progress bar more realistic?

2

remove and re-add the general repository and it will be fixed if you’re in Julia 1.6 or newer.

1 Like
3

A bit tangential and not julia specific, but no - that’s a REALLY hard problem, especially if the remaining work is not easily relatable to a timespan.

4

I removed ~/.julia/registries/General/ directory, and it indeeded sped up the fetching process a lot. Could you explain the reason for the speedup?

I notice that adding a package after removing the General/ directory creates General.tar.gz and General.toml. Maybe these tarball and .toml are a more efficient representation of the previous General/ directory?

5

In version 1.6 (I think) we switched the General repository from a git clone to a tarball, which especially on windows is a lot faster since windows gets really slow when you have lots of small files (and also this way the history of the registry isn’t stored). For complicated reasons, we by default didn’t change to the new method if you still had the old registry format downloaded (I think to prevent from breaking stuff for people that had set a local package server or something), so it requires the user to delete the old one. We really should have published this more broadly since it still effects a non-zero amount of people.

2 Likes
6

I think so. Depending on the network, fetching took >10 minutes for me, so I tended to postpone adding packages until leaving the computer for a while, e.g., for a lunch break. Now the fetching-induced delay is gone, so I can add packages whenever I want, which will increase my productivity.

2 Likes
7

This issue is that some people want to use git for this for various reasons and it’s not possible to distinguish someone who has a git repo because they just upgraded Julia and someone who created the git repo on purpose after upgrading.

8

The creation time of a file can be queried on Windows, OSX, and some Linux filesystems. Couldn’t we just issue a “consider upgrading your registry” message if the git repo predates v1.6, or if stale pre-v1.6 Julia environments exist in .julia/environments?

2 Likes