More problems trying to add packages from private repos

General Usage
#1

There have been many threads about the difficulties of using private repositories with Pkg. Here’s one, which contains links to various other threads:

At some point in the past I managed to get things working somehow, but now they appear to be broken again and I don’t know how to fix it. I decided to throw in the towel and connect to one of my private repos with HTTPS instead of SSH. And yet I still can’t actually add the repo. Here’s what happens:

(PackageA) pkg> add https://github.com/CameronBieganek/PackageB.jl.git
    Updating git-repo `https://github.com/CameronBieganek/PackageB.jl.git`
Username for 'https://github.com': myemail@gmail.com
Password for 'https://myemail@gmail.com@github.com': 
Username for 'https://github.com' [myemail@gmail.com]: myemail@gmail.com
Password for 'https://myemail@gmail.com@github.com': 
Username for 'https://github.com' [myemail@gmail.com]: myemail@gmail.com
Password for 'https://myemail@gmail.com@github.com': 
ERROR: failed to fetch from https://github.com/CameronBieganek/PackageB.jl.git, error: GitError(Code:EUSER, Class:Callback, Aborting, user cancelled credential request.)

What am I doing wrong? Why is this so hard? For now I’m giving up and and making one of my private repos public.

1 Like
#2

Mandatory question: can you git clone the package outside of Julia?

If that works you may want to try the Julia 1.7 release candidate with JULIA_PKG_USE_CLI_GIT=true, see https://github.com/JuliaLang/julia/blob/release-1.7/NEWS.md#package-manager.

1 Like
#3

I don’t know why connecting with HTTPS doesn’t work, but some time ago I also experienced issues adding my private packages via SSH. I think my problem was I was generating an SSH key that was incompatible with LibGit2. Once I generated a compatible SSH key things worked seamlessly. (See https://github.com/GunnarFarneback/LocalRegistry.jl/blob/master/docs/ssh_keys.md for details.)

2 Likes
#4

Thanks for the responses. Generating a new SSH key with the -m PEM option seemed to do the trick. Maybe when 1.7 is officially released I’ll use the JULIA_PKG_USE_CLI_GIT=true option.

1 Like
#5

Hi @GunnarFarneback, this is probably ignorant, but what does that actually mean? Where do I change the environment variable and then what external git executable do I use to download the private repository?

#6

This is a shell and operating system question.

In bash I would do:

export JULIA_PKG_USE_CLI_GIT=true
julia -t auto

In Windows Powershell,

$env:JULIA_PKG_USE_CLI_GIT=true
julia -t auto
#7

Thanks for both of your replies @mkitti I did this in bash, and now when I do Pkg.add(url=“…”) I get asked for the private key location, I enter the location and I receive the error: “failed to clone from git@github.com:placeholder.jl.git, error: GitError(Code:ERROR, Class:SSH, Failed to authenticate SSH session: Unable to open public key file”.

Thanks!

#8

Can you normally use git clone git@github.com:placeholder.jl.git?

If not then you may need to create a new key as indicated above, and then add that to github.

If you can clone it, then I recommend just giving Julia the local file path to the cloned repository.

#9

Those are all indications that Julia is not using CLI Git. What version of Julia are you using? What is the output of ENV["JULIA_PKG_USE_CLI_GIT"] in your running Julia?

#10

@mkitti I can, yes.

@GunnarFarneback, I created the startup file as instructed here. The following then happens… Thanks for your help!

julia> ENV["JULIA_PKG_USE_CLI_GIT"]
"true"

julia> using Pkg

julia> Pkg.add(url="git@github.com:users/packagename.jl.git")
ERROR: invalid git HEAD (reference 'refs/heads/master' not found)
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
  [2] check_valid_HEAD(repo::LibGit2.GitRepo)
    @ Pkg.GitTools /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/GitTools.jl:329
  [3] (::Pkg.Types.var"#42#43"{Pkg.Types.Context, Pkg.Types.PackageSpec, String})(repo::LibGit2.GitRepo)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:677
  [4] with(f::Pkg.Types.var"#42#43"{Pkg.Types.Context, Pkg.Types.PackageSpec, String}, obj::LibGit2.GitRepo)
    @ LibGit2 /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/LibGit2/src/types.jl:1150
  [5] handle_repo_add!(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:676
  [6] handle_repos_add!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:743
  [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:248
  [8] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
 [10] #add#26
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
 [11] add
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
 [12] add(; name::Nothing, uuid::Nothing, version::Nothing, url::String, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:166
 [13] top-level scope
    @ REPL[2]:1

caused by: GitError(Code:EUNBORNBRANCH, Class:Reference, reference 'refs/heads/master' not found)
Stacktrace:
  [1] macro expansion
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/LibGit2/src/error.jl:110 [inlined]
  [2] head(repo::LibGit2.GitRepo)
    @ LibGit2 /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/LibGit2/src/reference.jl:44
  [3] check_valid_HEAD(repo::LibGit2.GitRepo)
    @ Pkg.GitTools /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/GitTools.jl:327
  [4] (::Pkg.Types.var"#42#43"{Pkg.Types.Context, Pkg.Types.PackageSpec, String})(repo::LibGit2.GitRepo)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:677
  [5] with(f::Pkg.Types.var"#42#43"{Pkg.Types.Context, Pkg.Types.PackageSpec, String}, obj::LibGit2.GitRepo)
    @ LibGit2 /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/LibGit2/src/types.jl:1150
  [6] handle_repo_add!(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:676
  [7] handle_repos_add!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:743
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:248
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
 [10] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
 [11] #add#26
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
 [12] add
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
 [13] add(; name::Nothing, uuid::Nothing, version::Nothing, url::String, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:166
 [14] top-level scope
    @ REPL[2]:1
#11

I receive the following message when I try to download packages associated with a registry now too:

Pkg.add("Distributions")
    Updating registry at `~/.julia/registries/General.toml`
┌ Warning: could not download https://JULIA_PKG_USE_CLI_GIT=true/registries
│   exception = Could not resolve host: JULIA_PKG_USE_CLI_GIT=true while requesting https://JULIA_PKG_USE_CLI_GIT=true/registries
└ @ Pkg.Registry /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.7/Pkg/src/Registry/Registry.jl:82
   Resolving package versions...
#12

Did you set JULIA_PKG_SERVER by accident at some point?

Could you give us the output of versioninfo() within the REPL?

1 Like
#13

Could you show us the contents of startup.jl?

Usually you would want to set environment variables before starting Julia. It’s possible this will work though since this affects the package Pkg rather than julia itself.

1 Like
#14

@mkitti thanks for your help! I uninstalled Julia and VScode and reinstalled the program and I’m successfully connecting to the private repository. I’m sorry, I’m not sure what went wrong, but I set the startup file first, made sure the environment variable worked, then added the package… regardless, thanks for all your help!

#15

In some cases setting the environment variable in startup.jl may be too late. I’m still curious what the current output of versioninfo() is since I suspect your startup.jl may have been retained despite reinstall.

#16

Here you go!

Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, westmere)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 
  JULIA_PKG_SERVER = JULIA_PKG_USE_CLI_GIT=true
  JULIA_PKG_USE_CLI_GIT = true

I could be wrong, but before I uninstalled I ran versioninf() and I don’t think the Environemnt component was showing…

#17

Do you know where the below is coming from?

  JULIA_PKG_SERVER = JULIA_PKG_USE_CLI_GIT=true

Could you show us your startup.jl?

#18

My startup.jl only contains: ENV[“JULIA_PKG_USE_CLI_GIT”]=true

#19

What is modifying JULIA_PKG_SERVER? This is a big problem. Check you shell configuration or maybe start a fresh shell.

#20

Try running

unset JULIA_PKG_SERVER

in your shell. Hopefully it will not show up in versioninfo() when you start Julia again.