Julia and Gitlab self-hosted : a state-of-the-art?

General Usage
, ,
6

The setup at my company is less complex and correspondingly less ambitious in most, but not all, aspects.

  • Packages

Normally one per GitLab project (i.e. repository). Some projects contain dual Julia and Python packages, making use of the Pkg subdir functionality to point to the julia subdirectory. One project contains three closely related Julia packages.

If you have existing packages in git repositories elsewhere they are easily imported to GitLab. All git services have excellent instructions for how to import git repositories from anywhere else.

  • Testing

We run tests for all new commits (including merge requests and after merging to master) but no periodical tests of unchanged code. All tests are run in docker containers, either built from a Dockerfile in the repository or using a stock docker image from a common CI project. The typical test job is

test julia:
  stage: test
  script:
    - julia --project=julia -e 'using Pkg; Pkg.test()'

(This is an example from a dual package, so the Julia package is in the julia directory. The docker image is implicit from a default directive.)

  • Documentation

Most packages are documented by the README and additional markdown files in the repository. Some projects use an in-house documentation system which converts markdown files and various data sources to PDF files. This is just another CI job and the built PDFs are exported as CI artifacts.

  • Registry

The registry is just another repository. If you already have one on disk you can import to GitLab like any other repository. You should only need to update the repo field in the Package.toml for each package.
CI runs for all commits and basically consists of calling RegistryCI.test.

  • Registration

All julia packages use a common registration job, included from the common CI project, like this:

stages:
  - julia registration

include:
  project: $ci_project
  file:
    - templates/julia_package_registration.yml

variables:
  julia_package_dir: julia

The template is somewhat complex:

variables:
  Registry: https://project_71_bot:$JULIA_REGISTRATOR_TOKEN@$URL_TO_REGISTRY
  julia_package_dir: .
  package_repo_git_url: "git@$GITLAB_URL:$CI_PROJECT_PATH.git"

julia package registration:
  stage: julia registration
  only:
    - master
  script:
    - git config --global user.name $GITLAB_USER_NAME
    - git config --global user.email $GITLAB_USER_EMAIL
    - julia -e 'using Pkg; Pkg.add("LocalRegistry")'
    - cd $julia_package_dir
    - julia -e "using LocalRegistry; register(registry = \"$Registry\", repo = \"$package_repo_git_url\", create_gitlab_mr = true, ignore_reregistration = true)"

This uses an access token (the project_71_bot:$JULIA_REGISTRATOR_TOKEN stuff) to obtain credentials for the registry project and is run on every commit to master (but not on merge requests). If the version number has not been bumped, the LocalRegistry.register call won’t do anything. Merge requests for the registry are created by the create_gitlab_mr option and as soon as the registry CI job has passed they are automatically merged and immediately available.

  • Package Server

Packages are distributed using an in-house package server driven by the LocalPackageServer package. This means that the users have to set the environment variable JULIA_PKG_SERVER to point to this package server but they don’t have to add the company registry by URL, and they don’t need to have or bother with credentials for the GitLab server.

12 Likes