Verify generated images during automated testing

I am looking for a way to verify images that are auto-generated during CI/automated testing. For example, I would like to run a simulation, plot the result with Plots.jl or Makie.jl, and then verify the resulting PNG file. I have found this percentage difference script, but I was wondering if there is a package readily available that does such things in a smarter way. Some features that would also be nice:

  • Compare a generated image against a reference image but allow for small differences due to different rendering backends (i.e., Linux vs. macOS vs Windows)
  • Allow to test various properties in a convenient way (e.g., image size matches, color space matches etc.)
  • Calculate something like an “approximate” hash that can be used to compare generated images against a has value, without having to store the reference as an image file.

It may not do everything you’re hoping for, but GitHub - JuliaTesting/ReferenceTests.jl: Utility package for comparing data against reference files</title may contain some useful stuff.

You could try Percy, which is a commercial solution we use at my work but it has a free plan of 5k images per month, and has a nice polished UI for looking at visual diffs and hooking into GitHub so you can approve visual changes and have that reflect as a CI check passing on the PR. We use it in Lighthouse.jl, and just save out images to a folder during testing and then upload them to Percy in a later CI step.

Also, Makie.jl has its own vendored version of ReferenceTests.jl but I don’t know what the relation is to the JuliaTesting one.

Thanks a lot for your replies! They confirm my fear that there is no “proper”, ready-to-use package available, but the mentioned GitHub - JuliaTesting/ReferenceTests.jl: Utility package for comparing data against reference files</t package seems like the best option for now.