ANN: Automatic test-case reduction

testing
debug

#1

Hi all,

C-Reduce is a tool that takes a large C, C++, or OpenCL file that has a property of interest (such as triggering a compiler bug) and automatically produces a much smaller C/C++ file that has the same property.

I’ve used C-Reduce for reducing complicated Julia test-cases in the past (eg #28595 and #26278), and figured it might be useful for other people. A repo with instructions and support scripts is up at maleadt/creduce_julia, and depends on Julia 1.0+ and C-Reduce 2.8.0+ (both being available on PATH).

Basically, the process boils down to:

  • provide a script that has a certain property (eg. it crashes) in main.jl
  • edit run to catch that property (eg. julia main.jl |& grep "he's dead, jim")
  • run reduce
$ cat main.jl
error("example error message")

$ tail -n1 run
julia --compiled-modules=no -O0 main.jl |& grep "example error message"

# verify the `run` script works properly (should initially return 0)
$ ./run
ERROR: LoadError: example error message
$ echo $?
0

$ ./reduce
Reducing the following sources:
- main.jl
===< 7281 >===
...
===================== done ====================

If your script depends on certain packages, you can install them in the depot directory: Start by sourcing the activate script, which sets JULIA_DEPOT_PATH and JULIA_PKG_DEVDIR. You can now install packages which will end up in the depot directory, and will get picked up by the reduce script:

$ cat main.jl
using Compat
error("example error message")

$ source activate
$ julia
$ julia -e 'using Pkg; Pkg.add("Compat")'

# speed-up the process by removing useless files
$ rm -rf depot/packages/**/test

$ ./reduce
Reducing the following sources:
- main.jl
- depot/packages/Compat/HVYNa/src/Compat.jl
- depot/packages/Compat/HVYNa/src/arraymacros.jl
- depot/packages/Compat/HVYNa/src/compatmacro.jl
- depot/packages/Compat/HVYNa/src/deprecated.jl
===< 12424 >===
...
===================== done ====================

The process uses all available processors, but can still take a while (obviously depending on the run-time complexity, but think hours for single-file test-cases and days for multi-package setups). I’ve also used the process in conjunction with automated test-case generation through fuzzing, see this blog post for details.


Unnecessary compilation happening on every call