# Privately hosting BinaryBuilder products with authentication
**URL:** https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409
**Category:** Package Management
**Tags:** pkg, binarybuilder, artifacts
**Created:** [October 7, 2022, 3:43pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409 "2022-10-07T15:43:02Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [October 7, 2022, 3:43pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/1 "2022-10-07T15:43:02Z")
I’m trying to host some binaries I created with BinaryBuilder on my work’s internal web server. Accessing the server requires authentication.
I can use `curl` at the command line to download an artifact. By providing my username, I can get `curl` to prompt me for my password.
```julia
curl -O -u dmatz -L SOME_ARTIFACT_URL
Enter host password for user 'dmatz':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 243k 100 243k 0 0 338k 0 --:--:-- --:--:-- --:--:-- 346k
```
However, when I try to use the JLL package created by BinaryBuilder, with the same artifact URL, I get a 401 response without any opportunity to enter my password.
Is there a way to get Pkg to prompt me for my password? Is anyone else hosting BinaryBuilder products privately?
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [October 13, 2022, 1:26pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/2 "2022-10-13T13:26:29Z")
If you have a [netrc file](https://everything.curl.dev/usingcurl/netrc), Pkg (and any other usage of Downloads) will respect it.
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [October 13, 2022, 5:54pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/3 "2022-10-13T17:54:22Z")
Thank you! That did indeed work for me!
Does Pkg support any other methods of authentication besides a password? I’m learning from our sys admins that they will soon have to disable password authentication for all of our servers…
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [October 27, 2022, 6:58pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/4 "2022-10-27T18:58:49Z")
I realized I was suffering from some tunnel vision by only looking at HTTP options. Since I want to host these for internal use, I can just put them on our compute cluster and use SCP or SFTP to download them. That way we can use SSH keys to authenticate.
The issue now is that it is a bit tricky to get `curl` or `Downloads.download` to download a file from our server. I can easily download a file using `scp`:
```bash
$ scp my_server:.bashrc test.bashrc
.bashrc 100% 401 10.8KB/s 00:00
```
I can’t do it with `curl` like this:
```bash
$ /usr/local/Cellar/curl/7.85.0/bin/curl --verbose scp://my_server//home/dmatz/.bashrc
# cutting out some details
* SSH authentication methods available: publickey
* Using SSH private key file '/Users/dmatz/.ssh/id_rsa'
* SSH public key authentication failed: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format
* No identity would match
* Authentication failure
* Closing connection 0
curl: (67) Authentication failure
```
It seems I have to specify my username to get it to work:
```bash
$ /usr/local/Cellar/curl/7.85.0/bin/curl --verbose scp://my_server//home/dmatz/.bashrc --user dmatz:
```
If I then give `Downloads.download` on Julia 1.8.2 a try, I find that this doesn’t work:
```julia
julia> import Downloads
julia> Downloads.download("scp://my_server//home/dmatz/.bashrc", verbose = true)
# cutting out some details
* SSH authentication methods available: publickey
* Using SSH public key file '/Users/dmatz/.ssh/id_rsa.pub'
* Using SSH private key file '/Users/dmatz/.ssh/id_rsa'
* SSH public key authentication failed: Username/PublicKey combination invalid
* No identity would match
* Authentication failure
* Closing connection 0
```
Again, it seems it needs my username:
```bash
julia> Downloads.download("scp://dmatz@my_server//home/dmatz/.bashrc", verbose = true)
```
Is there some other option we can add to the libcurl config to get `Downloads.download` to work a bit better for SCP and SFTP?
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [November 5, 2022, 9:12pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/5 "2022-11-05T21:12:00Z")
How else would you like it to work? Specify the username as a keyword?
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [November 6, 2022, 2:39am UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/6 "2022-11-06T02:39:20Z")
It looks like `CURLOPT_USERNAME` can be used with scp for downloads.
[https://curl.se/libcurl/c/CURLOPT\_USERNAME.html](https://curl.se/libcurl/c/CURLOPT_USERNAME.html)
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [November 6, 2022, 6:11am UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/7 "2022-11-06T06:11:50Z")
> [@danielmatz](#):
>
> Is there some other option we can add to the libcurl config to get `Downloads.download` to work a bit better for SCP and SFTP?
You could try defining the host in your SSH config with your username set there for the host. I’m not 100% sure, but I think this should be picked up.
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [November 6, 2022, 7:48pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/8 "2022-11-06T19:48:21Z")
> [@mkitti](#):
>
> It looks like `CURLOPT_USERNAME` can be used with scp for downloads.
>
> [CURLOPT\_USERNAME](https://curl.se/libcurl/c/CURLOPT_USERNAME.html)
I was thinking the same thing at first. I even have a modified version of Downloads.jl that adds a keyword argument for the username and sets `CURLOPT_USERNAME`, which does indeed allow me to authenticate with SSH keys. However, I don’t see how this would be useful with artifacts. How would we allow users to set their own username? And for multiple servers?
What I was really trying to show is that SSH authentication in `libcurl` for some reason _requires_ you to set the username. Yet SSH authentication works fine for `ssh`, `scp`, and other command line utilities.
> [@Sukera](#):
>
> You could try defining the host in your SSH config with your username set there for the host. I’m not 100% sure, but I think this should be picked up.
This is also what I’ve been thinking. It seems like `libcurl` _should_ honor my SSH config, but it doesn’t. Even the `curl` command line utility isn’t honoring it, so it’s not just something we are configuring incorrectly…
Digging into the `libcurl` change log, I see that there are quite a few bug fixes related to their interaction with `libssh2` in versions that are newer than the one we are currently using. I also see some open issues on Yggdrasil to upgrade to a newer version of `libcurl`. Perhaps this will work better with a newer version?
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [November 6, 2022, 7:51pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/9 "2022-11-06T19:51:32Z")
It seems like curl doesn’t honor the SSH config file after all. See: [curl with sftp does not honor Port in ssh config file · Issue #9285 · curl/curl · GitHub](https://github.com/curl/curl/issues/9285).
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [November 6, 2022, 9:02pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/10 "2022-11-06T21:02:30Z")
We compile curl against LibSSH2
>
We probably need to refer to the LibSSH2A documentation for an external config file.
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [November 7, 2022, 1:59pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/11 "2022-11-07T13:59:40Z")
Ah, I see. So curl can be compiled with one of several ssh backends. It looks like libssh does support the SSH config file, but libssh2 does not. And, unfortunately, I don’t see any discussion in the libssh2 documentation of an alternate config file.
So, if we want to support SSH key authentication, I guess I’m back to figuring out a way to set `CURLOPT_USERNAME` in an automated fashion. Should we try to parse the SSH config file ourselves, ignoring most keys, but grabbing things like usernames and SSH key paths?
My overall goal here was to explore how I can host artifacts with authentication. It seems like there’s really only one option so far, which is to use the http method with a username and password in the `.netrc` file. Are there any other options I’m missing?
---
### Author:  [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)
#### Post date: [March 13, 2023, 8:54pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/12 "2023-03-13T20:54:43Z")
Any news about this ? I’m trying to host artifacts on a GitLab self-hosted instance with only ssh access, but so far, I did not find a way to retrieve them…
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [March 14, 2023, 1:19am UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/13 "2023-03-14T01:19:19Z")
I’ve been continuing to think about it, and I was hoping to put together a proposal and a PR soon, but I’ve been on paternity leave and just haven’t gotten around to it, yet.
I think we need to add a way for users to specify a configuration for Downloads.jl. We need this because libssh2 doesn’t honor the `.ssh/config` file, and libcurl doesn’t have a config file of its own (well, there is the `--config` option and the curlrc files, but they don’t let you specify options based on the host). I found a [discussion](https://curl.se/mail/lib-2022-08/0006.html) about SSH config files and the need for a better curl config file on the curl mailing list last year, and that would be wonderful if that happens some day, but it doesn’t help us in the near term.
I think we could keep it pretty simple. For a specified host and protocol, we could support setting the username, password, and paths to the SSH public and private keys. I’m not sure whether it would be with a config file or just a global config object that people can modify in their `startup.jl` files.
In the meantime, @BambOoxX, I’ve been downloading the artifacts manually and creating a depot that folks can point to using `JULIA_DEPOT_PATH`. Since they don’t need to download anything, they don’t need to authenticate! This is on our compute cluster with a shared file system.
---
### Author:  [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)
#### Post date: [March 14, 2023, 8:53am UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/14 "2023-03-14T08:53:52Z")
Correct me if I’m wrong, but in my case, I host the package on our private GitLab, as well as some dependencies. Therefore, both pulling a package and pulling dependencies need the same type of identification to reach the data on the server.
I would have expected both to succeed or fail, but it is not the case. Is it because pulling the package is done via `git` (which honors) ssh configuration, and pulling the dependencies via `curl` ?
---
### Author:  [@danielmatz](https://discourse.julialang.org/u/danielmatz)
#### Post date: [March 14, 2023, 2:57pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/15 "2023-03-14T14:57:18Z")
Right. For git repos, we use libgit2, and for artifacts, we use libcurl.
I generally use an SSH agent to get authentication to work for libgit2. I see that libcurl also supports using an SSH agent, but in practice I haven’t been able to get it to work. I think it’s related to setting the username, which I discussed above. If we could get it to work, perhaps we wouldn’t even need to add a way to configure Downloads.jl.
---
### Author:  [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)
#### Post date: [April 5, 2023, 8:53am UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/16 "2023-04-05T08:53:38Z")
FYI, it seems that the GitLab API does not support .netrc identification, which of course you cannot do anything about, but is it possible to use other identification processes in julia ? See also [How to configure an artifact to use a header during package download? - #2 by BambOoxX](https://discourse.julialang.org/t/how-to-configure-an-artifact-to-use-a-header-during-package-download/96057/2)
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [April 5, 2023, 7:54pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/17 "2023-04-05T19:54:56Z")
Whatever GitLab is doing is not a standard HTTP authentication mechanism, so I’m not sure what can be done here. They’re using a bespoke header to pass a secret token, which Downloads does allow you to do. What else would you want?
---
### Author:  [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)
#### Post date: [April 5, 2023, 8:03pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/18 "2023-04-05T20:03:01Z")
I do not want to digress to much here with respect to the original post, but I think what I am lacking in my case is simply how to configure an Artifact during a package development so that `Downloads` uses this header during package installation for the Artifact download. We can discuss this further in my own post if you prefer.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [April 5, 2023, 8:23pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/19 "2023-04-05T20:23:26Z")
You don’t configure the artifact, you would add a download hook that looks at the URL for a download and if the host name is [gitlab.com](http://gitlab.com), then it inserts the given header. I think that should be possible.
---
### Author:  [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)
#### Post date: [April 18, 2023, 4:31pm UTC](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409/21 "2023-04-18T16:31:34Z")
> [@StefanKarpinski](#):
>
> that looks at the URL for a download and if the host name is [gitlab.com](http://gitlab.com)
Please correct me if I’m wrong but when you say that it looks like something should be configured on the package user side for this solution to work, is that so ?
Also, I have no problem downloading something through `Downloads`, it is really about being able to use the same approach with artifacts. Basically being able to download manually an artifact should work, I just do not know if it’s possible.
[Next page](https://discourse.julialang.org/t/privately-hosting-binarybuilder-products-with-authentication/88409.md?page=2)