I am looking for a way to verify images that are auto-generated during CI/automated testing. For example, I would like to run a simulation, plot the result with Plots.jl or Makie.jl, and then verify the resulting PNG file. I have found this percentage difference script, but I was wondering if there is a package readily available that does such things in a smarter way. Some features that would also be nice:
Compare a generated image against a reference image but allow for small differences due to different rendering backends (i.e., Linux vs. macOS vs Windows)
Allow to test various properties in a convenient way (e.g., image size matches, color space matches etc.)
Calculate something like an “approximate” hash that can be used to compare generated images against a has value, without having to store the reference as an image file.
You could try Percy, which is a commercial solution we use at my work but it has a free plan of 5k images per month, and has a nice polished UI for looking at visual diffs and hooking into GitHub so you can approve visual changes and have that reflect as a CI check passing on the PR. We use it in Lighthouse.jl, and just save out images to a folder during testing and then upload them to Percy in a later CI step.
Thanks a lot for your replies! They confirm my fear that there is no “proper”, ready-to-use package available, but the mentioned https://github.com/JuliaTesting/ReferenceTests.jl package seems like the best option for now.