Hey, I’m a dev at Invenia (co-worker of @oxinabox), and this is an overview of our CI setup.
Every Julia package we have lives in its own GitLab repo. Each has a .gitlab-ci.yml file which defines the pipeline for that repo.
The one you linked to is a good starting point .gitlab-ci.yml · master · GitLab-examples / julia · GitLab.
The basic idea for the test job is to use a julia docker image, clone the repo and run Pkg.test.
Like in that example, you can have one job per julia version you want to support.
If you have many packages like we have at invenia, you may want to have a central yml file somewhere (we have this live in its own separate repo) which defines a standard pipeline, and then have all package repos include that file.
We run tests on all packages every night to see if updated dependencies break our code (they get updated according to the semver we specify in our Project.toml files).
We do this with a scheduled pipeline on each repo.
Now to get the tests to actually run you need to set up some machines with the Gitlab Runner installed. We do this on a Mac mini which we have setup in our office, and also on AWS, where we have an AutoScalingGroup that scales the number of machines up and down based on CPU usage. We test on different architectures, and do that with the use of GitLab CI tags. Can provide more details if needed.
Documentation is simply another job in the standard pipeline.
Actually, unlike the example you pointed to, we split it into two jobs.
“Documentation” builds the documentation (for every run of the pipeline, including for merge requests, so that we can review the doc changes before we approve and merge things), and “pages” publishes the docs (only on the default branch).
Here’s roughly what the .gitlab-ci.yml looks like for those jobs (untested, copy pasted and simplified stuff from different files, as our setup has complexified over the years):
"Documentation":
artifacts:
paths:
- documentation/
script:
julia --project=docs/ -e "using Pkg; Pkg.instantiate()"
julia --project=docs/ docs/make.jl
# Move the rendered documentation to a folder called "documentation" in the root of
# the repo which will be saved in an artifact.
mkdir documentation
mv docs/build/* documentation/
# Use the special job name "pages" to actually trigger deployment of the documentation on master.
# https://docs.gitlab.com/ee/user/project/pages/getting_started_part_four.html#job
pages:
only:
variables:
# Only deploy docs on the default branch (eg master)
- $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME
dependencies:
- Documentation
artifacts:
# As documentation is re-deployed every night the expiry just ensures that old documentation
# is eventually cleaned up.
expire_in: 1 week
paths:
- public/ # Note: Required to be called public for GitLab Pages
script:
- mkdir public
- '[ -d documentation ] && mv documentation/* public/'
We have a PackageRegistry repo, which is a private equivalent of the Julia General registry, with just our private packages in it.
(Not sure how this was setup initially).
Users should install this registry on their local machine in addition to the general one (replace with your URL / name):
git clone <your_registry_repo> ~/.julia/registries/<company_name>
All julia package repos have a “Register” job which runs LocalRegistry.jl register on it: (again untested, copy pasted from different places)
“Register”:
rules:
# Attempt registration when Project.toml changes in the default branch
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changes:
- "Project.toml"
script:
# Set git config so the commits to the registry are from a given GitLab user
- git config --global user.email "<email>"
- git config --global user.name "<username>"
# Register
- julia -e 'using Pkg; Pkg.add(name="LocalRegistry", version="0.5.2")'
- julia --project -e "using LocalRegistry; register(<local_package_path>, registry = \"<your_registry_repo>\", repo = \"$CI_PROJECT_URL\", create_gitlab_mr = true)"
(Special thanks to @gunnarfarneback for implementing the create_gitlab_mr functionality after we raised an issue!)
This will create a merge request in the PackageRegistry to add/update the julia package.
The PackageRegistry repo has its own .gitlab-ci.yml pipeline which runs the RegistryCI tests:
“​​Registry Tests":
script:
- julia --project=.ci/ -e 'using Pkg; Pkg.instantiate()'
- julia --project=.ci/ --color=yes -e 'using RegistryCI; RegistryCI.test(registry_deps=["https://github.com/JuliaRegistries/General.git"])'
Once the tests pass, you can merge that MR, and from that point the new package/version will be available to users.
Note we automate the merge using GitHub - invenia/TagBotGitLab: Julia TagBot for GitLab which we run in an AWS Lambda (this also creates the GitLab git tags and release notes for each package). There might be simpler ways to automate the merge.
- Migration from a local storage
We haven’t done this so can’t attest to it. But I think it is mostly about creating projects in Gitlab (1 project = 1 repo) and uploading & changing remote for your git repos (including the private registry).
So that’s the basics of it. A CI setup can be as complex as you need it to be (and ours has definitely become more complex over the years, to account for our many use cases). Happy to share more or answer specific questions. We’re thinking of putting out some blog posts on this topic, but it might take us a while to do that.
