I’m trying to host some binaries I created with BinaryBuilder on my work’s internal web server. Accessing the server requires authentication.
I can use curl at the command line to download an artifact. By providing my username, I can get curl to prompt me for my password.
curl -O -u dmatz -L SOME_ARTIFACT_URL
Enter host password for user 'dmatz':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 243k 100 243k 0 0 338k 0 --:--:-- --:--:-- --:--:-- 346k
However, when I try to use the JLL package created by BinaryBuilder, with the same artifact URL, I get a 401 response without any opportunity to enter my password.
Is there a way to get Pkg to prompt me for my password? Is anyone else hosting BinaryBuilder products privately?
I realized I was suffering from some tunnel vision by only looking at HTTP options. Since I want to host these for internal use, I can just put them on our compute cluster and use SCP or SFTP to download them. That way we can use SSH keys to authenticate.
The issue now is that it is a bit tricky to get curl or Downloads.download to download a file from our server. I can easily download a file using scp:
I was thinking the same thing at first. I even have a modified version of Downloads.jl that adds a keyword argument for the username and sets CURLOPT_USERNAME, which does indeed allow me to authenticate with SSH keys. However, I don’t see how this would be useful with artifacts. How would we allow users to set their own username? And for multiple servers?
What I was really trying to show is that SSH authentication in libcurl for some reason requires you to set the username. Yet SSH authentication works fine for ssh, scp, and other command line utilities.
This is also what I’ve been thinking. It seems like libcurlshould honor my SSH config, but it doesn’t. Even the curl command line utility isn’t honoring it, so it’s not just something we are configuring incorrectly…
Digging into the libcurl change log, I see that there are quite a few bug fixes related to their interaction with libssh2 in versions that are newer than the one we are currently using. I also see some open issues on Yggdrasil to upgrade to a newer version of libcurl. Perhaps this will work better with a newer version?
Ah, I see. So curl can be compiled with one of several ssh backends. It looks like libssh does support the SSH config file, but libssh2 does not. And, unfortunately, I don’t see any discussion in the libssh2 documentation of an alternate config file.
So, if we want to support SSH key authentication, I guess I’m back to figuring out a way to set CURLOPT_USERNAME in an automated fashion. Should we try to parse the SSH config file ourselves, ignoring most keys, but grabbing things like usernames and SSH key paths?
My overall goal here was to explore how I can host artifacts with authentication. It seems like there’s really only one option so far, which is to use the http method with a username and password in the .netrc file. Are there any other options I’m missing?