Cannot download general registry due to a corporate antivirus

General Usage
, ,
1

A change in my corporate antivirus cause problem when I try to install new packages.
I tried to change JULIA_PKG_DOWNLOAD_TIMEOUT and Curl.CURLOPT_LOW_SPEED limits but it seems to have no effect:

julia> ENV["JULIA_PKG_DOWNLOAD_TIMEOUT"]
"2000"

julia> ENV["JULIA_PKG_D"]^C

julia> using Downloads, Downloads.Curl

julia> Downloads.EASY_HOOK[] = (easy, info) -> begin
           # Désactive le timeout de vitesse basse (low speed limit/time)
           Curl.setopt(easy, Curl.CURLOPT_LOW_SPEED_TIME, 0)
           Curl.setopt(easy, Curl.CURLOPT_LOW_SPEED_LIMIT, 0)
       end
[ Warning: CURLOPT_LOW_SPEED_TIME is defined in LibCURL and is not public in Downloads.Curl
[ Warning: CURLOPT_LOW_SPEED_LIMIT is defined in LibCURL and is not public in Downloads.Curl
#2 (generic function with 1 method)

(@v1.12) pkg> add General
  Installing known registries into `~/.julia`
ERROR: could not download https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/a569d1e036b51ee1f6cf569c1bf28d0df3095bd6 
Exception: RequestError: HTTP/2 200 (Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds) while requesting https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/a569d1e036b51ee1f6cf569c1bf28d0df3095bd6

julia> Downloads.EASY_HOOK[] = (easy, info) -> begin
           # Désactive le timeout de vitesse basse (low speed limit/time)
           LibCurl.Curl.setopt(easy, Curl.CURLOPT_LOW_SPEED_TIME, 0)
           LibCurl.Curl.setopt(easy, Curl.CURLOPT_LOW_SPEED_LIMIT, 0)
       end
#5 (generic function with 1 method)

(@v1.12) pkg> add General
  Installing known registries into `~/.julia`
ERROR: could not download https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/a569d1e036b51ee1f6cf569c1bf28d0df3095bd6 
Exception: RequestError: HTTP/2 200 (Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds) while requesting https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/a569d1e036b51ee1f6cf569c1bf28d0df3095bd6

(@v1.12) pkg>

Any hints ?

2

I am not familiar with that and I don’t know whether these environment variables are related, but the libuv threads are created before you set the environment variables in your example. So it could be possible (again: I don’t know) that the environment variables are checked before you set them. Have you tried setting them before you start the Julia process?

3

I suspect this isn’t an antivirus, but rather a firewall. I also suspect your corporate IT won’t take kindly to circumventing their measures.

4

I will try this tomorrow. The Julia env variable was already set before the Julia session. I wonder if env curl variables will have an effect on the Julia session.

5

One IT manager has set (temporarily) a bypass for the antivirus for julialang.org and julialang.net: The problem was solved instantly.

They are reluctant to set this Bypass permanently and ask me to set the Julia timeout to a high value (more than 30 minutes). It seems that the JULIA_PKG_DOWNLOAD_TIMEOUT had no effect. Since the problem arise for the registry update it seems logical. I cannot find the solution…

6

I don’t understand how changing the download timeout or speed thresholds will help you if your network is blocking access in the first place.

Edit: oh, I get it — they’re doing some on-demand scanning that drastically slows things down?

8

yes

9

Could you try doing the download directly? That might help narrow in on the problem. For example, Downloads.jl includes the following example:

downloader = Downloads.Downloader()
downloader.easy_hook = (easy, info) -> Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_LOW_SPEED_TIME, 0)

Downloads.download("https://httpbingo.julialang.org/delay/30"; downloader)

You can replace the URL with the actual registry – https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/a569d1e036b51ee1f6cf569c1bf28d0df3095bd6. You could then potentially try to see if there’s an equivalent httpbingo endpoint that could replicate the issue outside of your network — maybe the drip endpoint?

10

What I’ve seen can happen is the first time you hit a url a rule can block you from accessing it until the file has been security scanned. When that happened, I just had to retry several times to get the script to ‘touch’ each url. Once whitelisted, the specific versions of dependencies were fast to download.