Creating a custom registry

General Usage
1

I’m trying to create a registry. I’m starting with a simple example before I attempt the real thing. My goal here is to make a registry with two packages TestB which depends on TestA which is also in the registry. Then I want to install TestB and see that it can find and download TestA. I’m using GunnarFarneback’s registrator version which allows the use of registrator for local actions, though I’m not 100% sure I’m using it correctly. I believe I created the two packages and the registry, but I can’t figure out how to make one package depend on the other. Also I haven’t got the the point of putting either package on github.

I’m going to paste the full list of commands I ran with some annotation, and then ask my question.

(v1.1) pkg> registry status
Registry Status 
 [23338594] General (https://github.com/JuliaRegistries/General.git)
(v1.1) pkg> add https://github.com/GunnarFarneback/Registrator.jl
using Registrator
shell> cd julia
shell> mkdir reg_test
shell> cd reg_test # I'm going to put the registry and both packages in this dir for the test
julia> create_registry("TestRegistry","https://github.com/ggggggggg/TestRegistry")
# make a github repo at that url
shell> cd TestRegistry/
shell> git push -u -f origin master # -f generally not neccesary, I wrote over my first attempt
shell> cd ..
# /Users/user/Julia/reg_test
(v1.1) pkg> generate TestA
(v1.1) pkg> generate TestB
shell> cd TestA
(v1.1) pkg> activate .
julia> using TestA # register requires module as argument, must using TestA 
shell> cd ..
# /Users/user/Julia/reg_test
(TestA) pkg> add TestRegistry # oops... I guess I added the registry as a package? it seems like that should have failed
(TestA) pkg> rm TestRegistry
(TestA) pkg> activate .
(reg_test) pkg> registry add TestRegistry
(TestB) pkg> activate TestB
# at this point I believe TestA exist, and is in TestRegistry, and I have added TestRegistry, so I should be able to find TestA from TestB... but clearly not
(TestB) pkg> add TestA
[ Info: resolving package specifier `TestA` as a directory at `~/Julia/reg_test/TestA`.
   Cloning git-repo `TestA`
ERROR: Git repository not found at 'TestA'
(TestB) pkg> registry status # it knows about TestRegistry, so why doesn't this work?
Registry Status 
 [23338594] General (https://github.com/JuliaRegistries/General.git)
 [0652d6e2] TestRegistry (https://github.com/ggggggggg/TestRegistry)

I’m not sure what I’m doing wrong, but I guess it has something to do with where my packages and directories are in my filesystem?

2

Since nobody with better expertise has replied, here are my 2 cents:

The best guide for creating a registry that I have seen is here.

I have tried this path somewhat unsuccessfully. I ended with a registry and entries in it that looked valid, but led to TOML errors that I could not parse.

When I try to replicate your steps, I get stuck at create_registry which has no binding in my setup.

One way of figuring out what may have gone wrong is to simply look at the registry files (that should be located in .jula/registries.

Hope this is somewhat helpful and curious to see how this gets resolved.

3

Thanks for trying to help. I think you may have missed the using Registrator step when following my steps.

@GunnarFarneback thanks for the package, do you any insight here?

4

It seems that you forgot the register step for TestA. The registry is, at this point, empty.

(I managed to get create_registry to work. Oddly, it failed a few times in spite of issuing using Registrator; I have no idea why).

5

I did leave out register by accident, but it was failing as well because it assumes your package is in a git repo. Once I did git init inside TestA then I was able to register and get add TestA to work inside TestB. I’ll try to start from scratch again and see where I end up.

6

I would be curious about the syntax for register. From the source, it is not clear to me what exactly is expected / required.

7

This works for me!! I wrote it all with function calls so it’s easier to copy/paste. I recommend either making your test repos public, or switching to git style urls so you can use ssh and a key. I typed my github login/password MANY times working this out. Also I think you can probably drop the repo= keyword arguments, but I’m not sure. I and I think you can drop at least the final Pkg.update(). Note that I use git push -f for the first round of pushes, since I was restarting this process multiple time.

Also it would be nice to figure out how to add to the registry in .julia/registries directly so I don’t need to working copies.

# make repos TestRegistry, TestA and TestB on github
import Pkg
Pkg.add(Pkg.PackageSpec(url="https://github.com/GunnarFarneback/Registrator.jl"))
import Registrator
mkdir("reg_test")
cd("reg_test")
Registrator.create_registry("TestRegistry","https://github.com/ggggggggg/TestRegistry")
run(`git -C TestRegistry push -u -f origin master`)
Pkg.generate("TestA")
run(`git -C TestA init`)
run(`git -C TestA add --all`)
run(`git -C TestA commit -m 'initial commit'`)
run(`git -C TestA remote add origin https://github.com/ggggggggg/TestA`)
run(`git -C TestA push -u -f origin master`)
Pkg.generate("TestB")
run(`git -C TestB init`)
run(`git -C TestB add --all`)
run(`git -C TestB commit -m 'initial commit'`)
run(`git -C TestB remote add origin https://github.com/ggggggggg/TestB`)
run(`git -C TestB push -u -f origin master`)
run(`git -C TestRegistry push origin master`)
Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/ggggggggg/TestRegistry"))
Pkg.activate("TestA")
import TestA
Registrator.register(TestA, "TestRegistry", repo="https://github.com/ggggggggg/TestA")
run(`git -C TestRegistry push origin master`)
Pkg.Registry.update()
Pkg.activate("TestB")
Pkg.add("TestA")
run(`git -C TestB add --all`)
run(`git -C TestB commit -m 'add TestA dep'`)
run(`git -C TestB push origin master`)
import TestB
Registrator.register(TestB, "TestRegistry", repo="https://github.com/ggggggggg/TestB")
run(`git -C TestRegistry push origin master`)
Pkg.Registry.update()



#testing that it works
exit()
cd ~
start julia
import Pkg
Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/ggggggggg/TestRegistry")) # would be required on fresh systems
Pkg.add("TestB")
2 Likes
8

Thanks! I have been unsuccessfully spending a fair bit of time on this.
Next step: How to push version updates…

9

I’m interested in what you figure out.

10

You have to push them manually: go to the local registry folder and call git push

11

Sorry - that was a bit cryptic for me.

My (untested) understanding is that one simply registers the same package again (after incrementing the version number in Project.toml).

Does your git push comment refer to what happens after that?

12

I think I misunderstood what you meant by your next step. Yes, you can add new versions to the registry the same way you add the first version (by calling register) along with an updated version number. My comment about the push was in reference to pushing a new version of the registry itself (e.g. to github). That branch of Registrator can’t do it for you.

13

Thanks - I hope to get around to trying this soon.

14

For anyone reading this now (Mar 24 2020): GunnarFarneback/Registrator ceases to work with Julia 1.4. Use LocalRegistry.jl instead.

1 Like
15

Yes, I have discontinued my Registrator fork in favor of the LocalRegistry package, which is now itself registered in the General registry. Since LocalRegistry is better tested, better documented, and is more convenient to use, I would recommend anyone who is still using my Registrator fork to switch, regardless of what Julia version you are running.

5 Likes
16

I am currently trying to use LocalRegistry and managed to

  • Create a new registry
  • Push it on a local NAS storage
  • Register packages into this registry
  • Push the modified registry

From there, I wanted to “share” this registry with some of my team members.
However, upon registry add <url_to_the_nas_registry_folder> it did not work at all, returning a git error.

My idea was that my teammates would only need to handle the registry in order to use / develop my packages, is this right or did i miss something ? For instance, is it reguired to use LocalPackageServer too ?

17

Yes, that’s the general idea.

For instance, is it reguired to use LocalPackageServer too ?

No, that’s not necessary.

From the error message in GitError(Code:ERROR, Class:Net, Unsupported URL protocol) · Issue #34 · GunnarFarneback/LocalRegistry.jl · GitHub I my best guess is that Pkg doesn’t parse

pkg> registry add \\nas_address\Git\Julia_Registry.git`

in the way you expect but I don’t have any Windows box to experiment with. What you could try is to use the functional form of registry add and explicitly tell it what path or URL you have.

help?> Pkg.Registry.add
  Pkg.Registry.add(url::String)
  Pkg.Registry.add(registry::RegistrySpec)

  Add new package registries.

  │ Julia 1.1
  │
  │  Pkg's registry handling requires at least Julia 1.1.

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  Pkg.Registry.add("General")
  Pkg.Registry.add(RegistrySpec(uuid = "23338594-aafe-5451-b93e-139f81909106"))
  Pkg.Registry.add(RegistrySpec(url = "https://github.com/JuliaRegistries/General.git"))

As a meta comment it’s usually better to start a new message/issue with your specific problem than to tag it onto some old conversation.

1 Like
18

I just figured people would stumble upon this thread (as I did) but I agree I could have started a new conversation. I am used to this on stackexchange, but discourse feels somewhat different…

I will test the functional version right now ! Thanks.

EDIT: The issue in my case was due to the non-logged state of the local user on the hosting NAS server preventing git to push/pull from it.
The Pkg.Registry.add(RegistrySpec(url=...)) syntax was also used to avoid any parsing issues.

1 Like