I am facing problems downloading artifacts on a linux remote server I connect to via SSH. This is an issue that I have been facing only in the last days and I am using this server daily since months now.
This is my versioninfo() output:
julia> versioninfo()
Julia Version 1.11.4
Commit 8561cc3d68d (2025-03-10 11:36 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 24 × Intel(R) Xeon(R) Platinum 8462Y+
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, haswell)
Threads: 24 default, 0 interactive, 12 GC (on 24 virtual cores)
Environment:
JULIA_PKG_USE_CLI_GIT = true
I am having issues downloading the General registry (get the warning below) but this has so far not caused specific error per se.
The real error happens when trying to download artifacts, and appeared first when trying to download XML2_jll package via indirect dependency, triggering a similar error for both the pkgserver and the github fallback:
I tried to pinpoint this issue, and I basically face the same problem when trying to directly download the tar.gz via Downloads.download, with the following output when doing verbose=true
julia> Downloads.download("https://github.com/JuliaBinaryWrappers/XML2_jll.jl/releases/download/XML2-v2.14.1+0/XML2.v2.14.1.x86_64-linux-gnu.tar.gz"; verbose = true)
* Couldn't find host github.com in the .netrc file; using defaults
* Host github.com:443 was resolved.
* IPv6: (none)
* IPv4: 140.82.121.3
* Trying 140.82.121.3:443...
* Connected to github.com (140.82.121.3) port 443
* mbedTLS: Connecting to github.com:443
* mbedTLS: Set min SSL version to TLS 1.0
* ALPN: curl offers h2,http/1.1
* Recv failure: Connection reset by peer
* ssl_handshake returned - mbedTLS: (-0x0001) ERROR - Generic error
* Closing connection
ERROR: RequestError: Recv failure: Connection reset by peer while requesting https://github.com/JuliaBinaryWrappers/XML2_jll.jl/releases/download/XML2-v2.14.1+0/XML2.v2.14.1.x86_64-linux-gnu.tar.gz
Strangely enough, when trying to directly download the same file on the shell using
I renamed the thread as the main issues seems to be really just on Downloads.download throwing an error on most of https link including github/pkg server which end up making Artifact downloads impossible on this remote machine.
I check possible previous threads with issues with Downloads like:
I have also seen that there used to be a way to override the downloader used by julia by setting the BINARYPROVIDER_DOWNLOAD_ENGINE variable but this does not seem to be applicable anymore in recent julia versions.
I initially though this issue might be temporary on the server I am working on but it’s now multiple days that is going on and it’s making my julia use on this linux host very problematic as a lot of packages now directly or indirectly rely on artifacts.
Is there a way to make julia use the system’s curl somehow for Artifacts (and or the whole Pkg stuff?) I doubt so but maybe worth asking.
Also shamelessly pinging @StefanKarpinski directly (sorry for the bother) as I see you were often answering questions with Downloads.download issues.
Do you get these problems with a completely fresh install? Specifically I am thinking of (temporarily) renaming your ~/.julia directory, reinstalling julia (with juliaup) and unsetting JULIA_PKG_USE_CLI_GIT and trying to install something from General registry. If that doesn’t work then try installing a package with the explicit URL, e.g. ] add https://github.com/JuliaLang/Example.jl. This will also test whether your problem will be solved by upgrading to 1.11.5
Thanks a lot for taking the time to reply here. Regarding your questions:
The server is inside my company’s network and for sure there is a firewall (not sure about transaprent proxy). I don’t know the exact details on the firewall per se. I do have access to a full shell on the server itself (but without admin rights) in case there is something I can run to help clarify this better
I installed julia with juliaup, so I do believe libcurl should be the one shipped with julia itself. I also never had issues before 5-6 days ago and nothing else changed on my side, so I assume some policy changes on firewall happened
As a side note, I also have a bunch of packages that are hosted on a company-wide gitlab instance hosted on company infrastracture. (Which I assume is subject to same or similar firewall policies).
Also on CI on this private gitlab instance, I do see the same problem on linux runners (but not on windows ones). And in that case I have CI jobs starting from the plain julia docker image.
Let me know if there is any command or futher investigation I might do to help diagnose the issue
A small update on this. I was able to override Pkg’s inner download function (thanks julia’s flexibility for this :D) to use the OS curl if it tries to download from https://pkg.julialang.org/
I do something very cursed (as per code on this gist) which is basically overwriting the function Pkg.PlatformEngines.download to substitute these lines:
into these other lines:
try
if startswith(url, "https://pkg.julialang.org")
# We override the default download command with this cursed function to use the OS's curl for requests to pkg.julialang.org
cmd = `curl -sS -L -o $dest`
for header in headers
key, val = header
push!(cmd.exec, "-H", "$key: $val")
end
push!(cmd.exec, url)
run(cmd)
else
Downloads.download(url, dest; headers, progress)
end
finally
By doing this, I confirm that I can install packages and Artifacts both on the server I connect to via SSH and also within our gitlab CI runners.
It is not pretty but at least allows me (and my colleagues) to keep working normally
I assume this means that firewall is not completely blocking links to pkgserver or github and hopefully it will be possible to find a cleaner solution for this (e.g. passing the correct parameters to libcurl used by Downloads.download)
So far it seems, I also had to add github among things to use OS curl for as some of our internal packages artifacts are there. But for what concerns artifacts that seems to solve. It would be nice to have something cleaner though
Hi @StefanKarpinski, one small clarifications on my previous message.
So far I only modified the codepath hit by the download of artifacts only. In reality most of codes relying on Downloads.download will be causing an error, so I think I should actually patch Downloads.download on the remote machine (and gitlab CI), though I really hope to find a solution where knowing which options to pass Libcurl would actually solve this issue. Alternatively, is there a rather simple way to override Downloads.download to use OS curl? I guess not really.
I just tried Downloads.download on some random https domains on the remote server with mixed results:
using Downloads
websites = [
"https://www.google.com",
"https://www.github.com",
"https://www.youtube.com",
"https://www.wikipedia.org",
"https://pkg.julialang.org",
"https://gitlab.com",
"https://www.docker.com",
"https://www.python.org",
"https://www.ubuntu.com",
"https://www.rust-lang.org",
"https://www.nodejs.org"
]
for website in websites
outcome = try
Downloads.download(website)
"OK"
catch e
"ERROR"
end
println("$website: $outcome")
end
which prints:
https://www.google.com: OK
https://www.github.com: ERROR
https://www.youtube.com: ERROR
https://www.wikipedia.org: ERROR
https://pkg.julialang.org: ERROR
https://gitlab.com: ERROR
https://www.docker.com: OK
https://www.python.org: OK
https://www.ubuntu.com: OK
https://www.rust-lang.org: OK
https://www.nodejs.org: OK
That looks an awful lot like a firewall enforcing a site blocking policy. The question is why command-line curl works. Maybe you could ask IT for some help understanding what’s going on? You could also ask them to allow HTTPS (port 443) traffic to julialang.org and julialang.net and subdomains.
I’m fairly sure that it’s not a MITM proxy issue because that would look like invalid certificate warnings rather than connection resets. My guess is that this is blocking by site and resetting connections to sites that are not allowed. The question then is: how is the firewall distinguishing the cli curl requests which it allow, from the Julia libcurl requests, which it blocks. If you don’t want to ask IT, another way to investigate would be to fire up WireShark and look at some traces comparing requests from the cli and Julia.
I tried to do some more digging (mostly with help of Claude as I am not very versed in wireshark and docker which was required to use wireshark as I don’t have sudo access on the machine).
One of the first things I noticed is that libcurl was using TLS 1.3, while download was using a lower version.
I tried going various hoops to force this in download without success, and another suggestions was that maybe the issue was caused by mbedTLS (while the OS version uses OpenSSL).
I have realized that more recent version of LibCURL_jll also switched to OpenSSL (based on the Project dependencies) and tried with julia 1.12 which indeed has a version of LibCURL which uses OpenSSL.
tried on julia 1.12 and indeed I don’t have problems with download there throwing errors so I guess this might be the actual issue? (Or any other modification between LibCURL_jll@v8.6 and LibCURL_jll@v8.11.1).
I guess this means I should try to switch everything on the server to julia 1.12 as soon as possible.
I guess there is no way to use OpenSSL on earlier julia versions right? Or no way to force LibCURL_jll to use the OS curl rather than its own?
Is this something that could have been changed on my company’s firewall side? (not accepting either mbedTLS or TLS < 1.3? Though it’s strange that some websites were still working correctly…
Oh that is curious. Glad it works on 1.12. If this is a legitimate issue, maybe we need to consider backporting OpenSSL usage to 1.10 since it’s the LTS release. It’s kind of a big change to backport but security fixes can be worth it.
In the end I found that the easiest solution is to simply overwrite the LibCURL_jll.jl entrypoint of the installed julia to put the full path of the system libcurl instead of the standard one.
This basically just boils down to substitute this line:
so that it points to "/lib/x86_64-linux-gnu/libcurl.so.4" rather than "libcurl.so.4" in my case.
This is very easy to do on the julia instalation for my user in the server and also very easy to do in CI.
This unfortunately does not work on julia < 1.11, probably because before 1.11 julia will always use compiled code from the system image rather than using the contents in the source code?
I had a quick check again at 1.11 release notes but didn’t find anything hinting at canges affecting this behavior.
I think I’ll still keep this as current solution as:
it works on CI and ssh connections,
is persistent (not in CI but that is not an issue),
most of our tests can skip 1.10
this solution also works in Pluto (as Pluto doesn’t really load startup file and would try to download packages before executing any package code, so there is no way of overriding the download function before it tries downloading package artifacts)
Just as extra information/curiosity on my side, is there anyway to use a different version of LibCURL_jll or in any way to change the libcurl library used internally by julia?
Regarding assessing whether this is a legitimate issue and not really just very specific to my server’s configuration, is there anything more I can do on my side to help debugging this @StefanKarpinski?
