Package update on Windows 7 and offline use

Running ]update on a fresh installation (no .julia folder) of Julia 1.5.1 just took me 35 minutes. Windows defender real-time protection was disabled. Observed this behavior on two separate Win 7 machines.

Connected with that issue, the difficulty of offline package installation is a glaring deficiency of Pkg.jl. Perhaps working without connectivity looks like an exotic and rare use case to someone with background in web development, but it is frequently necessary. It may be an organizational policy, or a machine may be interfacing some rare hardware and custom drivers could be damaged by the uncontrolled Windows 10 updates.

The ā€œalways-onā€ assumption also excludes cost-sensitive users, to whom a lab of a dozen machines all downloading the same set of packages is a significant drain on resources.

[Previously, I was having trouble downloading from eu-central.pkg.julialang.org, no longer hanging or aborting at 99%, maybe problem wasnā€™t server side. Based on your name Iā€™m not sure if you use eu-central, what do you connect to? Is it for you maybe a temporary problem, possibly neither a fault of julia?]

Running ]update on a fresh installation (no .julia folder) of Julia 1.5.1 just took me 35 minutes.

Thatā€™s very strange, for me on Linux (and I assume should be the experience for all including on Windows) I estimate about 10 sec. and the download itself 1.4 sec (I was very curious, as with a truly empty folder I wasnā€™t sure what update would even do):

$ mv ~/.julia .julia_bak
$ ~/julia-1.5.1/bin/julia -q

(@v1.5) pkg> update
 Installing known registries into `~/.julia`
--2020-08-31 13:01:16--  https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab
Resolving pkg.julialang.org (pkg.julialang.org)... 151.101.62.217
Connecting to pkg.julialang.org (pkg.julialang.org)|151.101.62.217|:443... connected.
HTTP request sent, awaiting response... 301 EU internal redirect trigger
Location: https://eu-central.pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab [following]
--2020-08-31 13:01:16--  https://eu-central.pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/81e1c37985c220a9a8052358e20d28c2cc94ffab
Resolving eu-central.pkg.julialang.org (eu-central.pkg.julialang.org)... 18.157.163.169
Connecting to eu-central.pkg.julialang.org (eu-central.pkg.julialang.org)|18.157.163.169|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2118082 (2,0M) [application/tar]
Saving to: ā€˜/tmp/jl_jtPV7U-download.gzā€™

/tmp/jl_jtPV7U-download.gz                100%[====================================================================================>]   2,02M  1,40MB/s    in 1,4s    

2020-08-31 13:01:17 (1,40 MB/s) - ā€˜/tmp/jl_jtPV7U-download.gzā€™ saved [2118082/2118082]

      Added registry `General` to `~/.julia/registries/General`
   Updating registry at `~/.julia/registries/General`
No Changes to `~/.julia/environments/v1.5/Project.toml`
No Changes to `~/.julia/environments/v1.5/Manifest.toml`

This 2 MB temp file is now gone, itā€™s not huge by todayā€™s standards, but Iā€™m still a bit curious about its contents.

In that scenario running a local copy of GitHub - JuliaPackaging/PkgServer.jl is worth investigating. If you want to also serve packages from a local registry, GitHub - GunnarFarneback/LocalPackageServer.jl: Julia storage and package server for local packages. can be of interest.

1 Like

I have the same experience as you on Linux, but not on Windows. The fault is not with server, because Iā€™ve had this issue on two different ISPs at different times, solely on Windows and never on Linux.

1 Like

Thank you, that might be worth looking into. What if I just want to install into a single unconnected laptop ?

I sometimes want update to be faster when .julia isnā€™t in fact empty, or e.g. in my test adding Plots.jl to an empty folder.

I want to ask was your .julia for sure empty? What I think might be happening, is similar to my test. You had an old 1.4 install. In my case I truly have an empty .julia and just added Plots.jl (to Julia 1.5.1), to have a non-empty folder for a previous version.

Then in a later version, Julia 1.6 I did:

(@v1.6) pkg> st
Status `~/.julia/environments/v1.6/Project.toml` (empty project)

Note, .julia isnā€™t empty, only that file. Maybe what you saw assuming .julia empty?

Then I did:

(@v1.6) pkg> update

but it was even quicker than with truly empty. Iā€™m just thinking could stuff from older version in the folder (e.g. if packages there not on latest version) interfere? [I just saw your answer now about only on Windows, it might be a bug, report it? or already done?]

Not just empty, but deleted, as in removed by hand

Strange. In Windows, where is the .julia folder? The . is a Unix notation to hide files/folders, is thatā€™s still the name on Windows?

On Windows, it is in ā€œC:\Users\Username\.juliaā€

Like I said please report it (canā€™t be intended behavior, and Iā€™m surprised nobody noticed), with the Windows label, with needed (at least minimal) info and link to this Discourse thread. [I hope they consider it, while Win7 no longer supportedā€¦ I doubt the issue is with Win7, but you never know, it might be related to something that doesnā€™t manifest on Win10.]

I scanned the Windows-labeled issues, and I donā€™t think itā€™s there but I did notice:
https://github.com/JuliaLang/julia/issues/36911

Anything in your setup unusual, like C:\Users on a slow network drive?

No, and as Iā€™ve mentioned, it behaved in the same way on laptop with Win 7 Home and desktop with Win 7 pro,

You didnā€™t answer that part, and is you ISP/country connection otherwise really slow? If you redo this, do you get similar to I did, only 2 MB file? And how long do that-size files usually take for you to download? Iā€™m not sure I can help more, Iā€™ve run out of ideas.

No, it is the same EU server, and it works fine when Linux is booted on the same machine, as Iā€™ve mentioned

Iā€™m not sure where to begin to debug the actual problem, I had a thought: julia --bug-report might help as ā€œTime Traveling (Linux) Bug Reportingā€ is really cool: Julia 1.5 Feature Preview: Time Traveling (Linux) Bug Reporting

But the rr tool it uses is only for Linux. On Linux I might use e.g. strace, but Iā€™m not sure whatā€™s a good substitute on Windows (itā€™s been long since I used).

A bit off-topic, someone is porting rr to Rust (while Iā€™m not clear itā€™s being ported to Windows, I doubt it): https://github.com/sidkshatriya/me/blob/master/002-why-rust.md

Can you elaborate? At some point you obviously need internet access to get the files to disk. Do you want to transfer files using a thumbdrive? Or can you connect the computer to install, and then work offline?

1 Like

Yes, transfer files using a USB drive

Check out my portable Julia/ VS Code. Build on one machine, transfer to another. Portable Julia - #8 by PetrKryslUCSD