Creating a registry

adamslc · July 2, 2018, 12:38am

I just spend quite a lot of time trying to create a custom registry for the new package manager, so I figured I would document my process so that it goes smoother for the next person.

Our goal is to create a registry TestRegistry that contains data about a single package TestPackage. We will suppose that TestPackage depends on Example.

Creating the package

First we create TestPackage, add Example as a dependency.

(v0.7) pkg> generate TestPackage
Generating project TestPackage:
    TestPackage/Project.toml
    TestPackage/src/TestPackage.jl

shell> cd TestPackage

(TestPackage) pkg> add Example
  Updating registry at `~/.julia/registries/TestReg`
  Updating git-repo `/Users/Luke/git/TestReg`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
  Updating `Project.toml`
  [7876af07] + Example v0.5.1
  Updating `Manifest.toml`
  [7876af07] + Example v0.5.1
  [8dfed614] + Test

We will push TestProject to some remote repo (Github in this case, but it could be anything in principle). Notice that we add Manifest.toml to the .gitignore file because we don’t want to make everyone use exactly our configuration.

cd TestPackage/
git init
echo /Manifest.toml > .gitignore
git add --all
git commit -m 'initial commit'
git remote add origin https://github.com/<username>/TestPackage.jl.git
git push -u origin master

Creating the registry

Now that we have our package, let’s create our registry. Return to your development directory, and create a new directory called TestRegistry. Then initialize a git repo.

mkdir TestRegistry
cd TestRegistry
git init

Now that we have created our registry, we need to fill it with the information about our package. Create the following folders and directories:

Registry.toml

We can use UUIDs.uuid1() to generate a uuid for the registry, and we will need to check TestPackage’s Project.toml file to find its uuid.

name = "TestRegistry"
uuid = "f1b7bfcd-108e-44d3-8130-d7177e4bc6a1"
repo = "https://github.com/<user>/TestRegistry.jl.git"
description = "This is only a test."

[packages]
28cd688e-7cf2-11e8-01a0-0dc1d6450aeb = { name = "TestPackage", path = "TestPackage" }

TestPackage/Package.toml

The Package.toml file gives basic data about TestPackage. It also specifies where to find the package.

name = "TestPackage"
uuid = "28cd688e-7cf2-11e8-01a0-0dc1d6450aeb"
author = "<your name here>"
repo = "https://github.com/<user>/TestPackage.jl.git"

TestPackage/Versions.toml

The Versions.toml file gives the git hash associated with every version. To add our first version (v0.1.0), return to the TestPackage directory and run git log to find the hash of the most recent commit.

["0.1.0"]
git-tree-sha1 = "3de6e7c41ca27d29cf976d127bf998556441d54d"

TestPackage/Deps.toml

In Deps.toml, we need to specify the name and uuid of the dependencies of TestPackage. We can lookup Example’s uuid, fill this in.

["0.1"]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"

TestPackage/Compat.toml

Finally, Compat.toml lists which versions our package is compatible with. Let’s suppose that our package only works with Julia version 0.7, and it can use either Example version 0.4 or 0.5.

["0.1"]
julia = "0.7"
Example = "0.4-0.5"

Now we will push all of this data to some remote repo (once again Github).

git add --all
git commit -m 'initial commit'
git remote add origin https://github.com/<user>/TestRegistry.jl.git
git push -u origin master

Using the registry

Now in .julia/registries/, we can clone the registry repo.

git clone https://github.com/<user>/TestRegistry.jl.git TestRegistry

If all has gone well, you should be able to fire up Julia, and add TestPackage.

(v0.7) pkg> add TestPackage
  Updating registry at `~/.julia/registries/TestRegistry`
  Updating git-repo `https://github.com/<user>/TestRegistry.jl.git`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
  Updating `~/.julia/environments/v0.7/Project.toml`
  [28cd688e] + TestPackage v0.1.0
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [7876af07] + Example v0.5.1
  [28cd688e] + TestPackage v0.1.0
  [8dfed614] + Test

Upgrading the package

Now suppose that we have added some sweet new features to TestPackage, and we would like to publish version 0.1.1. To do this, first edit Project.toml to bump the version number, and then commit and push this change.

vim Project.toml
git add Project.toml
git commit -m 'bump version to 0.1.1'
git push

Now we can update some files in TestRegistry to make this new version public and accessible to users.

TestPackage/Versions.toml

We use the output of git log to add a new entry to Versions.toml.

["0.1.0"]
git-tree-sha1 = "3de6e7c41ca27d29cf976d127bf998556441d54d"

["0.1.1"]
git-tree-sha1 = "3c67c429476fb35a1e86b23f7a5662beec2ab894"

TestPackage/Compat.toml

Let’s suppose that our new changes are not compatible with Example version 0.4.

["0.1"]
julia = "0.7"

["0.1.0"]
Example = "0.4-0.5"

["0.1.1"]
Example = "0.5"

Now we can push these updates, and Julia should automatically download the latest version of our registry.

git add --all
git commit -m "TestPackage v0.1.1"

Now we can update TestPackage in Julia.

(v0.7) pkg> up
  Updating registry at `~/.julia/registries/TestRegistry`
  Updating git-repo `https://github.com/<user>/TestRegistry.jl.git`
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
 Installed TestPackage ─ v0.1.1
  Updating `~/.julia/environments/v0.7/Project.toml`
  [28cd688e] ↑ TestPackage v0.1.0 ⇒ v0.1.1
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [28cd688e] ↑ TestPackage v0.1.0 ⇒ v0.1.1

I hope that this write up was helpful to someone. I tried to make it very detailed because I know that I appreciate it when documentation doesn’t make too many assumptions about what I know.

Nosferican · July 3, 2018, 8:55am

I believe the standard UUID version for the packages is UUID4 rather than UUID1, but can’t find the explicit reference in the documentation. Thanks for the basic rundown. If you manage to implement the scripts to update the registry systematically based on the Project.toml of a new release that would be great to have.

cstjean · August 20, 2018, 3:30pm

To get the tree-sha, use git log --pretty=format:'%T %s' instead of git log, which returns the commit SHA.

Ward9250 · December 7, 2018, 1:27am

I think they do use uuid1, it’s in the Pkg code currently: https://github.com/JuliaLang/Pkg.jl/blob/a17e84b1705d6f9e5c2489734b39423a1cc7ca1f/src/generate.jl#L51

Nosferican · December 12, 2018, 10:15pm

UUID4 used to be the default.

github.com

JuliaLang/Pkg.jl/blob/2075b74621f7829a930a480fe4c3b8c8f8ac0924/src/Operations.jl

# This file is a part of Julia. License is MIT: https://julialang.org/license

module Operations

using UUIDs
using Random: randstring
import LibGit2

import REPL
using REPL.TerminalMenus
using ..Types, ..GraphType, ..Resolve, ..Pkg2, ..BinaryProvider, ..GitTools, ..Display
import ..depots, ..depots1, ..devdir, ..Types.uuid_julia

function find_installed(name::String, uuid::UUID, sha1::SHA1)
    slug_default = Base.version_slug(uuid, sha1)
    # 4 used to be the default so look there first
    for slug in (Base.version_slug(uuid, sha1, 4), slug_default)
        for depot in depots()
            path = abspath(depot, "packages", name, slug)
            ispath(path) && return path

This file has been truncated. show original

fredrikekre · December 13, 2018, 12:08am

No? The 4 in the comment there refers to the slug-length.

Thuener · March 29, 2019, 1:27pm

In Julia 1.1 there is some suport on to add registry but create. There is any way to not do this by hand?

adamslc · March 29, 2019, 3:19pm

In short, no there isn’t, and Pkg will almost certainly never have that functionality built into it. There are however, some other options. If you are willing to set up some server infrastructure, then Registrator.jl should work for you. If not, then you can try to use this pull request for PkgDev.jl. Unfortunately, it looks like that PR has recently been closed without merging, which makes the future of small, privately managed registries uncertain.

StefanKarpinski · March 30, 2019, 1:22pm

There’s every intention to support small registries just not to have that extra functionality built into Pkg, which ships with every copy of Julia, whereas most people don’t need to create registries. As @kristoffer.carlsson indicated when he closed his PR, PkgDev is a legacy package, which is why it’s not being developed further. As he also said: Registrator is the future.

adamslc · March 30, 2019, 3:02pm

My concern is that Registraitor, in its current form, requires too much setup and infrastructure to be usable with small private repositories. I would much prefer something with a PkgDev like workflow. This issue is specifically what I’m thinking about.

To be clear, I think Registraitor will be an excellent solution for the General registry, but it doesn’t really scale to my use case well.

jpsamaroo · March 30, 2019, 6:41pm

Can I ask, for my own understanding, what the difficulties with Registrator are in its current form? I haven’t tried setting it up as of yet, but I do plan to do so to add support for another git forge I prefer to use. It would be good to know where the rough edges before I get started

ianfiske · April 1, 2019, 1:03pm

My initial concern is that Registrator appears to be Github-specific. If my organization uses another git hosting solution, privately hosted, can we still use some Registrator functions to create/manage a private registry of packages in our git platform?

StefanKarpinski · April 1, 2019, 1:43pm

The plan is to support other platforms besides GitHub as well but you’ve got to start somewhere and since most Julia packages are on GitHub currently, that’s the obvious place to start.

StefanKarpinski · April 1, 2019, 1:45pm

Factoring out the parts of the Registrator code base that would be useful standalone would be a fine project if you want to help contribute to that. What kind of a workflow did you have in mind?

ianfiske · April 1, 2019, 2:18pm

The Github-first approach totally makes sense given the Julia community’s distribution across platforms.

In the meantime, it seems like the core register function (https://github.com/JuliaComputing/Registrator.jl/blob/master/src/register.jl#L115) doesn’t depend on Github. Is there a plan to maintain this separation? If so, we can design around that.

adamslc · April 1, 2019, 3:27pm

I actually spent a few hours this weekend looking into the Registrator internals, and I will hopefully send a PR later this week. This seems like a good workflow:

julia> using MyPackage, Registrator

julia> Registrator.create_registry("MyRegistry")
Created an empty registry at ~/MyRegistry.

julia> Registrator.register(MyPackage, "~/MyRegistry")
Added package MyPackage.

julia> Registrator.register(MyPackage, "~/MyRegistry")
Error: Version 0.1 of MyPackage is already registered in ~/MyRegistery.
Perhaps you meant to update the version in Project.toml?

p.s. I’d like to apologize if I came off as too negative about Registrator.

StefanKarpinski · April 1, 2019, 6:31pm

Yes, it makes sense to keep it separate. I should probably just extract the part that just takes a new version and makes a new commit to a registry. Seems like that’s the thing people want as a standalone piece.

StefanKarpinski · April 1, 2019, 6:32pm

Yes that seems good. Do you have any opinion on whether it should be a separate package or just a local API exposed by Registrator?

adamslc · April 2, 2019, 8:47pm

I don’t have a preference either way. I have a draft PR to Registrator open here, but I’m happy to move the functionality elsewhere if you would prefer.

StefanKarpinski · April 2, 2019, 8:48pm

Ok, let’s not prematurely over-modularize. We can factor out the API for now and see how that goes. Installing some packages you don’t need isn’t all that horrible in the Pkg3 era.

Topic		Replies	Views
Creating a custom registry General Usage package-manager	17	3728	May 7, 2021
Switching package registration systems soon Announcements package-manager	172	14363	August 24, 2019
How to create a private package registry General Usage pkg	14	5061	July 10, 2019
Problem with reassignment of uuid when a package is registered Internals & Design packages	28	3480	June 10, 2018
Registering a package hosted on a public self-hosted gitlab instance New to Julia	13	1310	September 6, 2022

Creating a registry

Creating the package

Creating the registry

Registry.toml

TestPackage/Package.toml

TestPackage/Versions.toml

TestPackage/Deps.toml

TestPackage/Compat.toml

Using the registry

Upgrading the package

TestPackage/Versions.toml

TestPackage/Compat.toml

Related topics