Makie - How to add ReferenceTests when contributing to Makie

Specific Domains Visualization
1

I am not an experienced julia user, but I would like to contribute with a couple of minor implementations in Makie.jl. One of them is almost done, but I got stucked at one thing: I must implement a ReferenceTest.

This ReferenceTest is used to compare the output of a test code that produces an image with a given reference image.

Let us focus on my PR #4800 here, which simply allows the user to pass the alpha (transparency) parameter to a tricontourf plot.

I have already implemented the change in a git branch and commited a PR. Furthermore, I tried to add the new test (without running it locally), by introducing this code into Makie.jl / ReferenceTests / src / tests / examples2d.jl:

@reference_test "tricontourf alpha transparency" begin
    dxy = 1.0;
    x = [0.0, dxy, 0.0, -dxy, 0.0, dxy/2, -dxy/2, dxy/2, -dxy/2];
    y = [0.0, 0.0, dxy, 0.0, -dxy, dxy/2, dxy/2, -dxy/2, -dxy/2];
    @. f1(x,y) = x^2 + y^2;
    z = f1(x,y);
    
    f = Figure()
    ax1=Axis(f[1,1], title = "alpha = 1.0 (default)")
    ax2=Axis(f[1,2], title = "alpha = 0.5 (semitransparent)")
    hlines!(ax1, [-0.5, 0.0, 0.5])
    hlines!(ax2, [-0.5, 0.0, 0.5])
    tricontourf!(ax1, x, y, z, levels = 3)
    tricontourf!(ax2, x, y, z, levels = 3, alpha=0.5)
    f
end

I added test this and committed the PR to the official Makie repository on GitHub, but in fact I don’t know how to run this test locally.

When I send the PR, the test is failing because there is no reference image, and I also don’t know how to add this image and upload it.

Automatic testing fails because there is no reference image
Automatic testing fails because there is no reference image1086×639 33.5 KB

In Makie.jl / CONTRIBUTING.md, the following instruction is provided:

Please ensure locally that your feature works by running the tests. To be able to run the tests, you have to dev the helper package ReferenceTests that is part of the Makie monorepo. ReferenceTests is not a registered package, so you have to do ]dev path/to/ReferenceTests.

For my development, I am using these versions:

  • julia:1.11.3
  • Makie: 0.22.1
  • git: 2.34.1.windows.1
  • vscode: 1.97.2

I started my development like this:
] add Revise
] dev Makie
] dev CairoMakie

And then:

using Revise
using Makie
using CairoMakie

So I tried to add the package ReferenceTests.jl as follows:
]dev . ReferenceTests

In the documentation of ReferenceTests.jl, is says that the images with the missing references are created automatically. So I suppose I just have to run a test locally and the image will be created.

But I don’t have a folder with images in my locally cloned repo. When I check the source code of Makie.jl / .gitignore, I see that these paths are excluded and therefore not loaded into my computer:

WGLMakie/test/reference_images/
GLMakie/test/reference_images/
CairoMakie/test/reference_images/

That means, I won’t have the images of existing tests in my local repo, and if I create a new image, I think it wont be sent to GitHub.

Furtheremore, there is a file at ReferenceUpdater / src / image_download.jl. This file downloads or uploads the images as required, (or so I think).

So, here are my questions:

  1. What packages should I add and dev?
  2. If I add ReferenceTests.jl, should that be a subfolder of Makie.jl?
  3. Should I add] (or ]dev?) all Makie backends?
  4. How do I create and upload a reference image?
  5. There are many tests in examples2d.jl and I don’t want to run them all. How do I run locally only the reference test that I created?

Thank you in advance.

2

I think that documentation probably needs an update. For a basic PR that includes a new reference test, you basically just have to add the code creating a figure in one of the reference test files, wherever it fits best, give it a reasonable name, and test locally that if you run the code inside the macro, you get the figure you were expecting.

Then, when you make the PR, our CI will render the new image and create a note that a new reference image needs to be added. This will be done by a maintainer before merging the PR. You cannot upload a reference image, the set of reference images is managed by the maintainers because it is not part of the code and lives as a separate archive within the github repo under releases.

3

Thank you @jules. I marked the PR as completed, since the reference test had already been added, it was just the new reference image that was missing.

For more complex PRs, i.e. when I change existing functionality, I need to make sure that it doensn’t break other tests, or their reference images. I suppose it is then need it to run the tests locally, which I am not sure how to do. I’d appreciate some clear instructions on this.

Best regards.

4

To be honest, I don’t know because I always run the tests on CI and only test in a more ad-hoc kind of way locally. Maybe @sdanisch can elaborate on what the current workflow should be like for local tests