Yes, it is interesting. On the one hand, you’re treating your code like the subject of an experiment, like in the book The Design and Analysis of Computer Experiments. On the other hand, it can be as flatly wrong as an if-then missing an else. This thread has mentioned several approaches to minimize risk.
John’s “consistency” checking comes in many forms. You could ensure Lipshitz continuity of a result near a critical point, monotonicity, or a smooth approach to limiting values. There is a consistency check called metamorphic testing that looks for expected symmetries among arguments and results: given f(x)=y, find some g,h such that f(g(x))=h(y).
The “golden master” is also one of many kinds of checks against other implementations. You could compare Julia to an R implementation, a stochastic version to its continuum counterpart, your program’s answer to a paper’s graph, an inverse problem to its forward counterpart. Any of these place this work within context.
Colleagues and I struggle with testing well. It seems to be from a combination of missing tools, a lack of common practice, and a depth of knowledge required to be good at software, numerical analysis, and statistics, at the same time. I appreciate that you raised this question and enjoy hearing peoples’ approaches.