Using Pkg behind a firewall

question
#1

I’m trying to work with the Anaconda developers to iron out some problems with shared-library paths on MacOS (https://github.com/ContinuumIO/anaconda-issues/issues/6423), but they are seeing a weird error (with Julia 0.6.0 on MacOS):

julia> Pkg.init()
INFO: Initializing package repository /nfs/site/home/opavlyk/.julia/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:Zlib, error reading from the zlib stream)

I’ve never seen something like this, so it’s hard to help them; any pointers?

#2

It may, perhaps, be useful to know that I was behind a corporate firewall. Environment variables were set to make the command line git clone https://github.com/JuliaLang/METADATA.jl work correctly.

#3

Did you see https://docs.julialang.org/en/stable/manual/packages/#Packages-1 on using git config etc? Pkg uses libgit, not the git executable, so environment vars may not work the same way.

#4

I think I’m running into a similar issue running julia from within docker behind a corporate proxy. I have a dockerfile where I can install a bunch of packages, build julia from source, use curl, git clone, and similar from the command line, but can’t run Pkg.init() from within julia without getting the following error:

INFO: Initializing package repository /pkg/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:OS, Failed to connect to github.com: Connection refused)
Stacktrace:
 [1] macro expansion at ./libgit2/error.jl:99 [inlined]
 [2] clone(::String, ::String, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:276
 [3] #clone#100(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractCredentials}, ::Function, ::String, ::Str
ing) at ./libgit2/libgit2.jl:562
 [4] (::Base.LibGit2.#kw##clone)(::Array{Any,1}, ::Base.LibGit2.#clone, ::String, ::String) at ./<missing>:0
 [5] (::Base.Pkg.Dir.##8#10{String,String})() at ./pkg/dir.jl:55
 [6] cd(::Base.Pkg.Dir.##8#10{String,String}, ::String) at ./file.jl:70
 [7] init(::String, ::String) at ./pkg/dir.jl:53
 [8] init() at ./pkg/pkg.jl:85

I have HTTP_PROXY, HTTPS_PROXY, NO_PROXY, and the lowercase versions set, and also tried using the same settings in git config. On my development machine this works with just the environment variables set and no global .gitconfig.

#5

It’s also worth noting that the official Julia docker images work with only the environment variables defined – I can export JULIA_PKDIR='/some/other/folder' and then Pkg.init() works fine on both 0.6 and latest. I’m not sure why this is the case.

Edit: this may be related to building from source or running from within the build tree (as required by LLVM.jl) since I can build the official Dockerfiles fine just by adding in the environment variables.

#6

I think this issue can be closed. There were several local issues that I addressed. First, I needed to address SSL certificate problem.

Then, I ran into an obscure ERROR about “packfile trailer mismatch”, which was due to hitting home directory size quota while checking out METADATA.jl into ~/.julia/v0.6.

Once I symlinked ~/.julia to a local drive with no quota, things started to work as expected.

Thanks for looking at the issue though.