I have several dependencies from a private repository and I’m trying to figure out how to add them. Following another thread here on discourse, it seems I should add the leaf dependencies first and work my way up.
First, I’m having a problem where the private key never seems to work when used from Julia. I do
add private-git-url#master
at which point I get prompted for the location of the private key. If I enter a valid location, it prompts me several times (making the first value I input the default after the first time), but then returns an error
ERROR: failed to clone from private-git-url, error: GitError(Code:EAUTH, Class:Callback, Aborting, maximum number of prompts reached.)
The same key seems to work perfectly fine with git commands in both repositories.
My next question is, even if the private key issue can be resolved, will it ever be possible to do Pkg.instantiate() on my package? First, it has to know the location of the private keys (presumably beforehand, particularly if I execute instantiate from a script) and second, unless it clones them in precisely the right order it will throw a dependency error because the packages being cloned from the private repo depend on each other.
Any ideas on how to proceed?
In the meantime, it seems the alternative is just to clone the repos as part of a seperate process and add the local repos using Pkg.develop.
Keys: see How to specify the SSH key for Pkg to use.
Instantiate: In my experience that works fine even if you have dependent private packages.
Hm, that isn’t working. When I define those environment variables, I still get prompted for the key location, only now the correct location is listed as the default. Again, it prompts me 3 times, but eventually it just returns the same error.
Actually, keys are for the normal case easy, but you need to have a public key. I have
$ ls ~/.ssh
id_rsa id_rsa.pub known_hosts
and everything works smoothly without any prompting. If I move id_rsa.pub I get prompted about the private key (with correct default) and then about public key (with no default).
Instantiation from a Manifest doesn’t even call the resolver. It just replicates a Manifest.
1 Like
Yeah, the keys don’t seem to be working at all. They are indeed in ~/.ssh, I have both the private and public keys, they just are not called id_rsa and id_rsa.pub. I used the environment variables to list the locations of both the private and public keys but it still did not work.
Are you entirely certain that the public key matches the private key? If I corrupt my public key I get exactly your symptoms.
You can do ssh-keygen -y -f path_to_private_key to extract the public key from the private key.
Yup, did that, same problem.
Is it possible that there is some ridiculous security “feature” on the remote repo that’s preventing this from working correctly even though I have the right key? (Again, the keys work fine from git.)
If a git clone works with the same key, repository settings should be ok. There’s a chance you could be hit by a variation of https://github.com/JuliaLang/julia/issues/28933 but if I remember correctly the symptoms should be different.
I also tried commenting out the lines in my ~/.ssh/config that told it to use those keys for that particular server, but this had no effect.
Well, I’m really not at all sure what else to try here. I wouldn’t be at all surprised if it has something to do with how they have their git set up on the remote. Looks like I’m going to be stuck cloning with git and doing develop. That’s pretty unfortunate, since it’s going to make it much harder to reproduce this environment elsewhere since now the Manifest.toml will list explicit paths.
On a related note, is there anyway to get the Manifest.toml to use environment variables, such that the variables themselves appear in the actual Manifest.toml? Short of that, my only recourse will be to have multiple distinct Manifest.toml for builds in different locations.
If you have good enough relations with the remote repository admins, you could try to get hold of a server side ssh log, preferably at a decent level of verbosity.
If you want to test locally, you can use the technique in https://github.com/JuliaLang/julia/issues/28933#issuecomment-416731731
to see if you can get a working connection. If you want to test beyond the initial connection all you need is to place a bare git repository of your package somewhere and use an ssh url to that path, and you have your own remote repository.
If past events are any indication, the administrators of the remote repository likely have delusions that they work at NORAD and require absolutely top-notch security against well-funded opponents, even though in reality nothing that server is a steward of is in the slightest bit valuable to anyone not already using it. Suffice it to say, I don’t think they’ll be very eager to help me solve the issue.
Could you give me an example of how to use the bare repository? I’m imagining that I have to run some ssh command to expose the directory, but I’d probably have to do quite a bit of digging to figure out precisely what that is.
My other (admittedly very silly option) is to use symlinks to simulate a uniform environment.
So this is interesting, everything I’ve said so far was with the ssh URL (git@servername:location), if I instead use the HTTP url, it prompts me for a username and password. This seems to work (which was a big surprise to me, because I did not even think that server allowed cloning without RSA).
This doesn’t solve my problem however, since if I were to reproduce this on another environment in a dockerfile or whatever I wouldn’t be able to put in the username and password.
I must have hallucinated that I managed to do that from Pkg. I probably git cloned outside of Julia. Anyway, you don’t have to do anything to expose the directory. At the git level it looks like this:
gunnar@akka:~$ cd /tmp
gunnar@akka:/tmp$ git clone --bare https://github.com/GunnarFarneback/Inflate.jl.git
Cloning into bare repository 'Inflate.jl.git'...
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 62 (delta 28), reused 53 (delta 20), pack-reused 0
Unpacking objects: 100% (62/62), done.
gunnar@akka:/tmp$ mkdir test
gunnar@akka:/tmp$ cd test
gunnar@akka:/tmp/test$ git clone ssh://gunnar@localhost/tmp/Inflate.jl.git
Cloning into 'Inflate.jl'...
gunnar@localhost's password:
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 62 (delta 28), reused 0 (delta 0)
Receiving objects: 100% (62/62), 15.71 KiB | 7.86 MiB/s, done.
Resolving deltas: 100% (28/28), done.
gunnar@akka:/tmp/test$ ls
Inflate.jl
Oh, I jumped ahead, I was thinking what if I want to clone on another machine. Still, that was a stupid question because I’d suppose I’d just have to set up ssh like normal.
Suppose I want to use the username an password credentials, is there any way to put them in there so it doesn’t prompt for it when I do ]add?
Hello all, I just wanted to bump this. I still can’t get the RSA keys working for some reason, but I can indeed clone through HTTP with username and password. Does anyone know of any way I can get it to put them in there automatically so that I can instantiate in a build script with no user interaction?
1 Like
I had problems with rsa keys on Windows machine. I got an advice: make sure that git works in Power shell. Basically, this meant adding the git to the path and that solved my problem. (I don’t understand why because I thought julia uses some git library.)
Indeed, Julia uses libgit2. git works fine for me as it uses my .ssh/config. Again, even though libgit prompts me for the key locations it doesn’t not seem to want to accept them. I tried a few different permission configurations.
Yes exactly and after setting the path to Git for Windows bin the problem disappeared mystically.