Is there a way to prevent Julia from updating the registries almost every time?

[PetrKryslUCSD]

On Windows 10 it is possible to speed up the registry download by adding the .julia folder to the anti virus exclusion list: Why is the update of the registries so slow? - #11 by PetrKryslUCSD.

[joa-quim]

@GunnarFarneback

e.g. pkg> registry rm General .

But if we do that Julia versions lower than 1.7 won’t work since they don’t know about the tared registry, right?

[GunnarFarneback]

They don’t know about it, ignore it, and download registries the way they are used to. See Read the registry straight from the tarball by KristofferC · Pull Request #2431 · JuliaLang/Pkg.jl · GitHub.

[dlakelan]

@StefanKarpinski

Yikes. The things they call file systems these days.

Well, glusterfs has a lot of great things going for it. But it is slow when it comes to stat (listing files) or deleting things. Basically it’s got to maintain its redundant synchronization between the servers, and that takes overhead. It does seem like the stats are cached so the second time I traverse a filesystem tree it’s fast… but that first time is potentially really slow.

[dlakelan]

For reference. I did download 1.7 beta, and did

] registry rm General

which took maybe an hour or so, and then

] registry add General

which took maybe 1 second

[StefanKarpinski]

That’s great to hear! Lots of slow burn work went into making that happen and the diff functionality will make this even better.

[dlakelan]

You guys are amazing. I can’t express how fortunate I feel to have Julia. I donate to you guys every year because you are doing “gods work” so to speak. I only wish I had more time to use the dang thing in between walking my rescue dog, feeding my kids, and avoiding COVID. (but hey, when I do sit down and have time to work on a project, I really appreciate that adding a package doesn’t take 2.5 hours)

[joa-quim]

@dlakelan

For reference. I did download 1.7 beta, and did

] registry rm General

which took maybe an hour or so, and then

] registry add General

which took maybe 1 second

I did the same and second step took a couple seconds but the first one was very quick too (Windows here).

[HanD]

@HanD

@jdad

Just a further comment about size of registry transfers. You are talking about hundred of megabytes, which hints me that it is a Git tar that is downloaded (with 1.7, the tar is only 3MB). If you are on 1.6, you could try to change registry type by registry rm General / registry add General which should change the transfer to tar of data only, if I am not mistaken.

Thanks, @jdad, My registry copy was inherited from Julia v1.0, I wasn’t aware that this could cause an issue. I tried what you suggested, and I will check the network usage for future registry updates.

Sooo, I’m still unsure what the removal and the re-adding of the General registry exactly did to my local copy, but the above trick did indeed solve the issue. And not only it reduced the network usage from 100 MB per update to approx. 3 MB, it also significantly improved the update time.

(Note that I’m still running Julia 1.6.2, so it is surely not the mentioned unpacked storage of the registry.)