How can you guarantee that all the functions / branches / loops in your package has been tested?

General Usage
,
1

Is there any automated testing method for a rather complicated package? Probably a macro that injects some logging commands to each function / if-else branch / for-loop and produce a statistical analysis of the coverage of a testing procedure? I know the macro @profile can record execution time, but how about coverage ?

2

Coverage is usually handled by continuous integration tools, e.g., codecov. You can setup you github repo with automatic hooks to codecov such that you get a warning if code coverage decreased by pulling a PR. You can even prevet a PR from being merged unless codecov returned the desired status.

You find more info at
https://github.com/JuliaCI/Coverage.jl

3 Likes
3

Concerning the testing and coverage handled in Continuous Integration tools that @baggepinnen was referring to, you can have a look on this tutorial to learn how to applying it to Julia packages…

2 Likes
4

This is exactly what I wanted. Thanks !

5

The tutorial is very elaborate. Thank you very much!

6

However note that services like codecov only show you line code coverage, but tell you nothing about paths of the code: conditionals, loops, and other control flow statements can make the behaviour of a line very different if it’s hit from different places.

5 Likes
7

@giordano mentioned that there are different kinds of coverage. You have to figure out what arguments to the function will give you good coverage, and there are four ways to do that.

  1. Construct them by hand, as in a decision table.
  2. Randomly choose each argument from a set of values, including edge cases. Run the randomized tests for a while.
  3. Use an all-pairs strategy, from UnitTestDesign.jl.
  4. Use property-based testing, from randomizedpropertytest.jl.

Which method works best will depend on how long the function runs, how many arguments it has, and how much risk is involved. Good luck!

3 Likes
8

trying them out, thank you !

9

In line with Mosè Giordano, what is the point of guaranteeing that all functions / branches / loops in your package have been tested?

In the words of Dijkstra (1969): “Testing shows the presence, not the absence of bugs.”

EDIT: I mean, Dijkstra doesn’t say that testing is pointless, but the efforts spent on guaranteeing that everything is covered by tests is probably better spent on other verification techniques or refactoring and documenting.

2 Likes
10

Yes I agree. Now I realized that unit test + coverage doesn’t guarantee good code. I should simplify the internal logic of the code.

11

I wrote a package to do this exactly this kind of testing. It works as follows: it swallows the whole program, parses out all the conditionals, and then spits out the code again as a giant ‘eval’ with an invocation-counter-increment call attached to each conditional. Obviously, this makes execution slower. It was written for an old version of Julia, possibly 0.4, so most likely it won’t work any more but you are welcome to have a look. GitHub - StephenVavasis/microcoverage

2 Likes
12

OUAH !
Thank a lot !