Package update on Windows 7 and offline use

Running ]update on a fresh installation (no .julia folder) of Julia 1.5.1 just took me 35 minutes. Windows defender real-time protection was disabled. Observed this behavior on two separate Win 7 machines.

Connected with that issue, the difficulty of offline package installation is a glaring deficiency of Pkg.jl. Perhaps working without connectivity looks like an exotic and rare use case to someone with background in web development, but it is frequently necessary. It may be an organizational policy, or a machine may be interfacing some rare hardware and custom drivers could be damaged by the uncontrolled Windows 10 updates.

The “always-on” assumption also excludes cost-sensitive users, to whom a lab of a dozen machines all downloading the same set of packages is a significant drain on resources.

[Previously, I was having trouble downloading from eu-central.pkg.julialang.org, no longer hanging or aborting at 99%, maybe problem wasn’t server side. Based on your name I’m not sure if you use eu-central, what do you connect to? Is it for you maybe a temporary problem, possibly neither a fault of julia?]

Running ]update on a fresh installation (no .julia folder) of Julia 1.5.1 just took me 35 minutes.

That’s very strange, for me on Linux (and I assume should be the experience for all including on Windows) I estimate about 10 sec. and the download itself 1.4 sec (I was very curious, as with a truly empty folder I wasn’t sure what update would even do):

$ mv ~/.julia .julia_bak
$ ~/julia-1.5.1/bin/julia -q

(@v1.5) pkg> update
 Installing known registries into `~/.julia`
--2020-08-31 13:01:16--  https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab
Resolving pkg.julialang.org (pkg.julialang.org)... 151.101.62.217
Connecting to pkg.julialang.org (pkg.julialang.org)|151.101.62.217|:443... connected.
HTTP request sent, awaiting response... 301 EU internal redirect trigger
Location: https://eu-central.pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab [following]
--2020-08-31 13:01:16--  https://eu-central.pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab
Resolving eu-central.pkg.julialang.org (eu-central.pkg.julialang.org)... 18.157.163.169
Connecting to eu-central.pkg.julialang.org (eu-central.pkg.julialang.org)|18.157.163.169|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2118082 (2,0M) [application/tar]
Saving to: ‘/tmp/jl_jtPV7U-download.gz’

/tmp/jl_jtPV7U-download.gz                100%[====================================================================================>]   2,02M  1,40MB/s    in 1,4s    

2020-08-31 13:01:17 (1,40 MB/s) - ‘/tmp/jl_jtPV7U-download.gz’ saved [2118082/2118082]

      Added registry `General` to `~/.julia/registries/General`
   Updating registry at `~/.julia/registries/General`
No Changes to `~/.julia/environments/v1.5/Project.toml`
No Changes to `~/.julia/environments/v1.5/Manifest.toml`

This 2 MB temp file is now gone, it’s not huge by today’s standards, but I’m still a bit curious about its contents.

In that scenario running a local copy of GitHub - JuliaPackaging/PkgServer.jl is worth investigating. If you want to also serve packages from a local registry, GitHub - GunnarFarneback/LocalPackageServer.jl: Julia storage and package server for local packages. can be of interest.

1 Like

I have the same experience as you on Linux, but not on Windows. The fault is not with server, because I’ve had this issue on two different ISPs at different times, solely on Windows and never on Linux.

1 Like

Thank you, that might be worth looking into. What if I just want to install into a single unconnected laptop ?

I sometimes want update to be faster when .julia isn’t in fact empty, or e.g. in my test adding Plots.jl to an empty folder.

I want to ask was your .julia for sure empty? What I think might be happening, is similar to my test. You had an old 1.4 install. In my case I truly have an empty .julia and just added Plots.jl (to Julia 1.5.1), to have a non-empty folder for a previous version.

Then in a later version, Julia 1.6 I did:

(@v1.6) pkg> st
Status `~/.julia/environments/v1.6/Project.toml` (empty project)

Note, .julia isn’t empty, only that file. Maybe what you saw assuming .julia empty?

Then I did:

(@v1.6) pkg> update

but it was even quicker than with truly empty. I’m just thinking could stuff from older version in the folder (e.g. if packages there not on latest version) interfere? [I just saw your answer now about only on Windows, it might be a bug, report it? or already done?]

Not just empty, but deleted, as in removed by hand

Strange. In Windows, where is the .julia folder? The . is a Unix notation to hide files/folders, is that’s still the name on Windows?

On Windows, it is in “C:\Users\Username\.julia”

Like I said please report it (can’t be intended behavior, and I’m surprised nobody noticed), with the Windows label, with needed (at least minimal) info and link to this Discourse thread. [I hope they consider it, while Win7 no longer supported… I doubt the issue is with Win7, but you never know, it might be related to something that doesn’t manifest on Win10.]

I scanned the Windows-labeled issues, and I don’t think it’s there but I did notice:

Anything in your setup unusual, like C:\Users on a slow network drive?

No, and as I’ve mentioned, it behaved in the same way on laptop with Win 7 Home and desktop with Win 7 pro,

You didn’t answer that part, and is you ISP/country connection otherwise really slow? If you redo this, do you get similar to I did, only 2 MB file? And how long do that-size files usually take for you to download? I’m not sure I can help more, I’ve run out of ideas.

No, it is the same EU server, and it works fine when Linux is booted on the same machine, as I’ve mentioned

I’m not sure where to begin to debug the actual problem, I had a thought: julia --bug-report might help as “Time Traveling (Linux) Bug Reporting” is really cool: https://julialang.org/blog/2020/05/rr/

But the rr tool it uses is only for Linux. On Linux I might use e.g. strace, but I’m not sure what’s a good substitute on Windows (it’s been long since I used).

A bit off-topic, someone is porting rr to Rust (while I’m not clear it’s being ported to Windows, I doubt it): https://github.com/sidkshatriya/me/blob/master/002-why-rust.md

Can you elaborate? At some point you obviously need internet access to get the files to disk. Do you want to transfer files using a thumbdrive? Or can you connect the computer to install, and then work offline?

1 Like

Yes, transfer files using a USB drive

Check out my portable Julia/ VS Code. Build on one machine, transfer to another. Portable Julia