# Verify generated images during automated testing

**URL:** <https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389>\
**Category:** General Usage\
**Tags:** testing, images\
**Created:** [August 14, 2021, 12:39pm UTC](https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389 "2021-08-14T12:39:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 14, 2021, 12:39pm UTC](https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389/1 "2021-08-14T12:39:34Z")

</div>

I am looking for a way to verify images that are auto-generated during CI/automated testing. For example, I would like to run a simulation, plot the result with Plots.jl or Makie.jl, and then verify the resulting PNG file. I have found this [percentage difference script](https://rosettacode.org/wiki/Percentage_difference_between_images#Julia), but I was wondering if there is a package readily available that does such things in a smarter way. Some features that would also be nice:

- Compare a generated image against a reference image but allow for small differences due to different rendering backends (i.e., Linux vs. macOS vs Windows)
- Allow to test various properties in a convenient way (e.g., image size matches, color space matches etc.)
- Calculate something like an “approximate” hash that can be used to compare generated images against a has value, without having to store the reference as an image file.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [August 14, 2021, 1:28pm UTC](https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389/2 "2021-08-14T13:28:46Z")

</div>

It may not do everything you’re hoping for, but [https://github.com/JuliaTesting/ReferenceTests.jl](https://github.com/JuliaTesting/ReferenceTests.jl) may contain some useful stuff.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 14, 2021, 6:15pm UTC](https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389/3 "2021-08-14T18:15:19Z")

</div>

You could try [Percy](http://percy.io/), which is a commercial solution we use at my work but it has a free plan of 5k images per month, and has a nice polished UI for looking at visual diffs and hooking into GitHub so you can approve visual changes and have that reflect as a CI check passing on the PR. We use it in [Lighthouse.jl](https://github.com/beacon-biosignals/Lighthouse.jl), and just save out images to a folder during testing and then [upload them to Percy in a later CI step](https://github.com/beacon-biosignals/Lighthouse.jl/blob/779c4b16324d574646e869e928e4eae48ceb5e16/.github/workflows/CI.yml#L40).

Also, Makie.jl has [its own vendored version of ReferenceTests.jl](https://github.com/JuliaPlots/Makie.jl/tree/master/ReferenceTests) but I don’t know what the relation is to the JuliaTesting one.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [August 19, 2021, 11:20am UTC](https://discourse.julialang.org/t/verify-generated-images-during-automated-testing/66389/4 "2021-08-19T11:20:25Z")

</div>

Thanks a lot for your replies! They confirm my fear that there is no “proper”, ready-to-use package available, but the mentioned [https://github.com/JuliaTesting/ReferenceTests.jl](https://github.com/JuliaTesting/ReferenceTests.jl) package seems like the best option for now.