Which protocols can be used for adding a package by Pkg add from a git server?

Internals & Design
1

Hello everyone,
I’m configuring a server that will be a host for our Julia packages. I have made a private Julia registry that our users will add to their registry, and everything has been sat up to work in our private network.
My question is when performing Pkg.add("Example.jl") Julia uses which git protocols to clone or get the packages. I have read the git-server documentation, and it seems that Julia doesn’t use the Dumb HTTP protocol.
I wanted to ask this before trying every git protocol.

Thank you very much for your help

2

Looking under ~.julia/registries/General/Registry.toml it has the line:

repo = "https://github.com/JuliaRegistries/General.git"

Which says to me that it’s using HTTPS to access github. Checking a few of the registered packages they also appear to use HTTPS. However I would assume if the GIT library understands any valid git URL that it could handle HTTP:// or git://.

3

I assumed that since Julia is using the LibGit2 library, it must support all git server types. But as I have mentioned, testing the ‘dumb HTTP’ protocol yield to error.
Let me show the error.
I put a bare clone of the Example.jl package on my server.
Julia add function result:

(v1.3) pkg> add http://myServer.com/julia/test/Example.jl.git/
   Cloning git-repo `http://myServer.com/julia/test/Example.jl.git/`
ERROR: failed to clone from http://myServer.com/julia/test/Example.jl.git/, error: GitError(Code:ERROR, Class:OS, failed to retrieve response content-type: The requested header was not found
)

git clone on the terminal is OK. The result is:

$ git clone http://myServer.com/julia/test/Example.jl.git/
Cloning into 'Example.jl'...

$ ls
Example.jl/

$ cd Example.jl\

$ ls
docs/  LICENSE.md  Project.toml  README.md  src/  test/

$ git remote -v
origin  http://myServer.com/julia/test/Example.jl.git/ (fetch)
origin  http://myServer.com/julia/test/Example.jl.git/ (push)

That behavior raised this question that maybe Julia is restricted to certain types of git server protocols.

4

Does LibGit2.clone work?

5

No the same Error:

julia> repo_url = "http://myServer.com/julia/test/Example.jl.git/"
julia> repo = LibGit2.clone(repo_url,"new\\")
ERROR: GitError(Code:ERROR, Class:OS, failed to retrieve response content-type: The requested header was not found
)
Stacktrace:
 [1] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\LibGit2\src\error.jl:101 [inlined]
 [2] clone(::String, ::String, ::LibGit2.CloneOptions) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\LibGit2\src\repository.jl:459
 [3] #clone#131(::String, ::Bool, ::Ptr{Nothing}, ::Nothing, ::Dict{Symbol,Tuple{Ptr{Nothing},Any}}, ::typeof(LibGit2.clone), ::String, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\LibGit2\src\LibGit2.jl:580
 [4] clone(::String, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\LibGit2\src\LibGit2.jl:559
 [5] top-level scope at REPL[12]:1
6

I found this old answer about LibGit2 library that says this library works with dumb HTTP, but I think the Julia implementation of LibGit2 may differ from the LibGit2 in C language.

7

From my googling it appears that HTTP is supported but “smart” HTTP whatever that is, not the “WebDAV/static” transport. Maybe this will help?

https://git-scm.com/book/en/v2/Git-on-the-Server-Smart-HTTP

1 Like
8

Ok. I must set up a smart HTTP git server and check whether ‘Pkg.add’ works or not. I think there is a good chance that this will work with smart HTTP, but why it doesn’t work with the dumb HTTP protocol? Considering that setting a dumb HTTP protocol is very easy, it may be helpful to have this option too.

9

No clue, my guess would be “testing surface area”. I’m a developer, our main product supports 3 different databases. We get requests every now and again to support some additional databases, but even though SQL is “generic” there are quirks and custom statements. SO we have to test on all databases we support before a release. Supporting more databases is not that attractive when it significantly increases our QA cycle.

I’m guessing the same thing is happening here, adding support for it would mean testing it for releases and if there is not a “large” demand for that protocol, they don’t want the overhead. Or since it’s an open source product maybe that protocol is not cool enough for anyone to get interested in just implementing it…

2 Likes
10

You can see the implementation of LibGit2.clone here: https://github.com/JuliaLang/julia/blob/b133e92904ed0e40761ee79c4db2fff3ae6eec6b/stdlib/LibGit2/src/repository.jl#L454
It’s a very thin wrapper around the call to the corresponding LibGit2 C function.

1 Like