# Pkg: private repository usage and keys

**URL:** <https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575>\
**Category:** General Usage\
**Tags:** pkg\
**Created:** [September 27, 2018, 5:05pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575 "2018-09-27T17:05:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 5:05pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/1 "2018-09-27T17:05:37Z")

</div>

I have several dependencies from a private repository and I’m trying to figure out how to add them. Following another thread here on discourse, it seems I should add the leaf dependencies first and work my way up.

First, I’m having a problem where the private key never seems to work when used from Julia. I do

```julia
add private-git-url#master

```

at which point I get prompted for the location of the private key. If I enter a valid location, it prompts me several times (making the first value I input the default after the first time), but then returns an error

```julia
ERROR: failed to clone from private-git-url, error: GitError(Code:EAUTH, Class:Callback, Aborting, maximum number of prompts reached.)

```

The same key seems to work perfectly fine with `git` commands in both repositories.

My next question is, even if the private key issue can be resolved, will it ever be possible to do `Pkg.instantiate()` on my package? First, it has to know the location of the private keys (presumably beforehand, particularly if I execute `instantiate` from a script) and second, unless it clones them in precisely the right order it will throw a dependency error because the packages being cloned from the private repo depend on each other.

Any ideas on how to proceed?

In the meantime, it seems the alternative is just to clone the repos as part of a seperate process and add the local repos using `Pkg.develop`.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 5:09pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/2 "2018-09-27T17:09:23Z")

</div>

Keys: see [How to specify the SSH key for Pkg to use](https://discourse.julialang.org/t/how-to-specify-the-ssh-key-for-pkg-to-use/14784).

Instantiate: In my experience that works fine even if you have dependent private packages.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 5:20pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/3 "2018-09-27T17:20:11Z")

</div>

Hm, that isn’t working. When I define those environment variables, I still get prompted for the key location, only now the correct location is listed as the default. Again, it prompts me 3 times, but eventually it just returns the same error.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 5:21pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/4 "2018-09-27T17:21:56Z")

</div>

Actually, keys are for the normal case easy, but you need to have a public key. I have

```julia
$ ls ~/.ssh
id_rsa id_rsa.pub known_hosts

```

and everything works smoothly without any prompting. If I move `id_rsa.pub` I get prompted about the private key (with correct default) and then about public key (with no default).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 27, 2018, 5:22pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/5 "2018-09-27T17:22:01Z")

</div>

> [@ExpandingMan](#):
>
> unless it clones them in precisely the right order it will throw a dependency error because the packages being cloned from the private repo depend on each other.

Instantiation from a Manifest doesn’t even call the resolver. It just replicates a Manifest.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 5:24pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/6 "2018-09-27T17:24:31Z")

</div>

Yeah, the keys don’t seem to be working at all. They are indeed in `~/.ssh`, I have _both_ the private and public keys, they just are not called `id_rsa` and `id_rsa.pub`. I used the environment variables to list the locations of _both_ the private and public keys but it still did not work.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 5:27pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/7 "2018-09-27T17:27:51Z")

</div>

Are you entirely certain that the public key matches the private key? If I corrupt my public key I get exactly your symptoms.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 5:32pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/8 "2018-09-27T17:32:01Z")

</div>

You can do `ssh-keygen -y -f path_to_private_key` to extract the public key from the private key.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 5:34pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/9 "2018-09-27T17:34:05Z")

</div>

Yup, did that, same problem.

Is it possible that there is some ridiculous security “feature” on the remote repo that’s preventing this from working correctly even though I have the right key? (Again, the keys work fine from `git`.)

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 5:39pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/10 "2018-09-27T17:39:25Z")

</div>

If a `git clone` works with the same key, repository settings should be ok. There’s a chance you could be hit by a variation of [https://github.com/JuliaLang/julia/issues/28933](https://github.com/JuliaLang/julia/issues/28933) but if I remember correctly the symptoms should be different.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 5:45pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/11 "2018-09-27T17:45:11Z")

</div>

I also tried commenting out the lines in my `~/.ssh/config` that told it to use those keys for that particular server, but this had no effect.

Well, I’m really not at all sure what else to try here. I wouldn’t be at all surprised if it has something to do with how they have their git set up on the remote. Looks like I’m going to be stuck cloning with git and doing `develop`. That’s pretty unfortunate, since it’s going to make it much harder to reproduce this environment elsewhere since now the `Manifest.toml` will list explicit paths.

On a related note, is there anyway to get the `Manifest.toml` to use environment variables, such that the variables themselves appear in the actual `Manifest.toml`? Short of that, my only recourse will be to have multiple distinct `Manifest.toml` for builds in different locations.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 6:01pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/12 "2018-09-27T18:01:29Z")

</div>

If you have good enough relations with the remote repository admins, you could try to get hold of a server side ssh log, preferably at a decent level of verbosity.

If you want to test locally, you can use the technique in [https://github.com/JuliaLang/julia/issues/28933#issuecomment-416731731](https://github.com/JuliaLang/julia/issues/28933#issuecomment-416731731)  
to see if you can get a working connection. If you want to test beyond the initial connection all you need is to place a bare git repository of your package somewhere and use an ssh url to that path, and you have your own remote repository.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 6:12pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/13 "2018-09-27T18:12:38Z")

</div>

If past events are any indication, the administrators of the remote repository likely have delusions that they work at NORAD and require absolutely top-notch security against well-funded opponents, even though in reality nothing that server is a steward of is in the slightest bit valuable to anyone not already using it. Suffice it to say, I don’t think they’ll be very eager to help me solve the issue.

Could you give me an example of how to use the bare repository? I’m imagining that I have to run some `ssh` command to expose the directory, but I’d probably have to do quite a bit of digging to figure out precisely what that is.

My other (admittedly very silly option) is to use symlinks to simulate a uniform environment.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 6:33pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/14 "2018-09-27T18:33:47Z")

</div>

So this is interesting, everything I’ve said so far was with the `ssh` URL (`git@servername:location`), if I instead use the HTTP url, it prompts me for a username and password. This seems to work (which was a big surprise to me, because I did not even think that server allowed cloning without RSA).

This doesn’t solve my problem however, since if I were to reproduce this on another environment in a dockerfile or whatever I wouldn’t be able to put in the username and password.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 7:29pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/15 "2018-09-27T19:29:16Z")

</div>

> [@ExpandingMan](#):
>
> Could you give me an example of how to use the bare repository? I’m imagining that I have to run some `ssh` command to expose the directory, but I’d probably have to do quite a bit of digging to figure out precisely what that is.

I must have hallucinated that I managed to do that from `Pkg`. I probably `git clone`d outside of Julia. Anyway, you don’t have to do anything to expose the directory. At the git level it looks like this:

```julia
gunnar@akka:~$ cd /tmp
gunnar@akka:/tmp$ git clone --bare https://github.com/GunnarFarneback/Inflate.jl.git
Cloning into bare repository 'Inflate.jl.git'...
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 62 (delta 28), reused 53 (delta 20), pack-reused 0
Unpacking objects: 100% (62/62), done.
gunnar@akka:/tmp$ mkdir test
gunnar@akka:/tmp$ cd test
gunnar@akka:/tmp/test$ git clone ssh://gunnar@localhost/tmp/Inflate.jl.git
Cloning into 'Inflate.jl'...
gunnar@localhost's password: 
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 62 (delta 28), reused 0 (delta 0)
Receiving objects: 100% (62/62), 15.71 KiB | 7.86 MiB/s, done.
Resolving deltas: 100% (28/28), done.
gunnar@akka:/tmp/test$ ls
Inflate.jl

```

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 27, 2018, 7:37pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/16 "2018-09-27T19:37:26Z")

</div>

Oh, I jumped ahead, I was thinking what if I want to clone on another machine. Still, that was a stupid question because I’d suppose I’d just have to set up `ssh` like normal.

Suppose I want to use the username an password credentials, is there any way to put them in there so it doesn’t prompt for it when I do `]add`?

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 29, 2018, 5:55pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/17 "2018-10-29T17:55:19Z")

</div>

Hello all, I just wanted to bump this. I still can’t get the RSA keys working for some reason, but I can indeed clone through HTTP with username and password. Does anyone know of any way I can get it to put them in there automatically so that I can instantiate in a build script with no user interaction?

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [October 29, 2018, 7:38pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/18 "2018-10-29T19:38:46Z")

</div>

I had problems with rsa keys on Windows machine. I got an advice: make sure that git works in Power shell. Basically, this meant adding the git to the path and that solved my problem. (I don’t understand why because I thought julia uses some git library.)

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 29, 2018, 7:42pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/19 "2018-10-29T19:42:04Z")

</div>

Indeed, Julia uses `libgit2`. `git` works fine for me as it uses my `.ssh/config.` Again, even though `libgit` prompts me for the key locations it doesn’t not seem to want to accept them. I tried a few different permission configurations.

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [October 29, 2018, 7:49pm UTC](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575/20 "2018-10-29T19:49:54Z")

</div>

Yes exactly and after setting the path to Git for Windows bin the problem disappeared mystically.

[Next page](https://discourse.julialang.org/t/pkg-private-repository-usage-and-keys/15575.md?page=2)