Creating a registry

package
tutorials

#1

I just spend quite a lot of time trying to create a custom registry for the new package manager, so I figured I would document my process so that it goes smoother for the next person.

Our goal is to create a registry TestRegistry that contains data about a single package TestPackage. We will suppose that TestPackage depends on Example.

Creating the package

First we create TestPackage, add Example as a dependency.

(v0.7) pkg> generate TestPackage
Generating project TestPackage:
    TestPackage/Project.toml
    TestPackage/src/TestPackage.jl

shell> cd TestPackage

(TestPackage) pkg> add Example
  Updating registry at `~/.julia/registries/TestReg`
  Updating git-repo `/Users/Luke/git/TestReg`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
  Updating `Project.toml`
  [7876af07] + Example v0.5.1
  Updating `Manifest.toml`
  [7876af07] + Example v0.5.1
  [8dfed614] + Test 

We will push TestProject to some remote repo (Github in this case, but it could be anything in principle). Notice that we add Manifest.toml to the .gitignore file because we don’t want to make everyone use exactly our configuration.

cd TestPackage/
git init
echo /Manifest.toml > .gitignore
git add --all
git commit -m 'initial commit'
git remote add origin https://github.com/<username>/TestPackage.jl.git
git push -u origin master

Creating the registry

Now that we have our package, let’s create our registry. Return to your development directory, and create a new directory called TestRegistry. Then initialize a git repo.

mkdir TestRegistry
cd TestRegistry
git init

Now that we have created our registry, we need to fill it with the information about our package. Create the following folders and directories:

Registry.toml

We can use UUIDs.uuid1() to generate a uuid for the registry, and we will need to check TestPackage's Project.toml file to find its uuid.

name = "TestRegistry"
uuid = "f1b7bfcd-108e-44d3-8130-d7177e4bc6a1"
repo = "https://github.com/<user>/TestRegistry.jl.git"
description = "This is only a test."

[packages]
28cd688e-7cf2-11e8-01a0-0dc1d6450aeb = { name = "TestPackage", path = "TestPackage" }

TestPackage/Package.toml

The Package.toml file gives basic data about TestPackage. It also specifies where to find the package.

name = "TestPackage"
uuid = 28cd688e-7cf2-11e8-01a0-0dc1d6450aeb
author = "<your name here>"
repo = "https://github.com/<user>/TestPackage.jl.git"

TestPackage/Versions.toml

The Versions.toml file gives the git hash associated with every version. To add our first version (v0.1.0), return to the TestPackage directory and run git log to find the hash of the most recent commit.

["0.1.0"]
git-tree-sha1 = "3de6e7c41ca27d29cf976d127bf998556441d54d"

TestPackage/Deps.toml

In Deps.toml, we need to specify the name and uuid of the dependencies of TestPackage. We can lookup Example's uuid, fill this in.

["0.1"]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"

TestPackage/Compat.toml

Finally, Compat.toml lists which versions our package is compatible with. Let’s suppose that our package only works with Julia version 0.7, and it can use either Example version 0.4 or 0.5.

["0.1"]
julia = "0.7"
Example = "0.4-0.5"

Now we will push all of this data to some remote repo (once again Github).

git add --all
git commit -m 'initial commit'
git remote add origin https://github.com/<user>/TestRegistry.jl.git
git push -u origin master

Using the registry

Now in .julia/registries/, we can clone the registry repo.

git clone https://github.com/<user>/TestRegistry.jl.git TestRegistry

If all has gone well, you should be able to fire up Julia, and add TestPackage.

(v0.7) pkg> add TestPackage
  Updating registry at `~/.julia/registries/TestRegistry`
  Updating git-repo `https://github.com/<user>/TestRegistry.jl.git`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
  Updating `~/.julia/environments/v0.7/Project.toml`
  [28cd688e] + TestPackage v0.1.0
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [7876af07] + Example v0.5.1
  [28cd688e] + TestPackage v0.1.0
  [8dfed614] + Test 

Upgrading the package

Now suppose that we have added some sweet new features to TestPackage, and we would like to publish version 0.1.1. To do this, first edit Project.toml to bump the version number, and then commit and push this change.

vim Project.toml
git add Project.toml
git commit -m 'bump version to 0.1.1'
git push

Now we can update some files in TestRegistry to make this new version public and accessible to users.

TestPackage/Versions.toml

We use the output of git log to add a new entry to Versions.toml.

["0.1.0"]
git-tree-sha1 = "3de6e7c41ca27d29cf976d127bf998556441d54d"

["0.1.1"]
git-tree-sha1 = "3c67c429476fb35a1e86b23f7a5662beec2ab894"

TestPackage/Compat.toml

Let’s suppose that our new changes are not compatible with Example version 0.4.

["0.1"]
julia = "0.7"

["0.1.0"]
Example = "0.4-0.5"

["0.1.1"]
Example = "0.5"

Now we can push these updates, and Julia should automatically download the latest version of our registry.

git add --all
git commit -m "TestPackage v0.1.1"

Now we can update TestPackage in Julia.

(v0.7) pkg> up
  Updating registry at `~/.julia/registries/TestRegistry`
  Updating git-repo `https://github.com/<user>/TestRegistry.jl.git`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
 Installed TestPackage ─ v0.1.1
  Updating `~/.julia/environments/v0.7/Project.toml`
  [28cd688e] ↑ TestPackage v0.1.0 ⇒ v0.1.1
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [28cd688e] ↑ TestPackage v0.1.0 ⇒ v0.1.1

I hope that this write up was helpful to someone. I tried to make it very detailed because I know that I appreciate it when documentation doesn’t make too many assumptions about what I know.


#2

I believe the standard UUID version for the packages is UUID4 rather than UUID1, but can’t find the explicit reference in the documentation. Thanks for the basic rundown. If you manage to implement the scripts to update the registry systematically based on the Project.toml of a new release that would be great to have.


#3

To get the tree-sha, use git log --pretty=format:'%T %s' instead of git log, which returns the commit SHA.