I’m using Julia to test the integrity of a large dataset by re-calculating values at random locations and checking that they match.
When a test that was run using the @test macro fails, it reports what was evaluated (and its stack trace), but I can’t work out how to provide additional context to those error messages—in my case, I’d like to report the location in the dataset that the error occurred.
Here’s a toy example to demonstrate this:
using Test
function foobar(n)
return n % 1000 == 0 ? true : false
end
@testset "Sample test" begin
for i = 1:10000
@test foobar(rand(1:1000)) == false
end
end
If I run the above it tells me that it fails some number of times. But the context in this example (analogous to the “location in the dataset”) would be the i value: what i value(s) did it fail at? Is there some way of being able to report this?
You can put @testset on the loop directly and interpolate the loop variable into the description string:
@testset "Sample test $i" for i = 1:10000
@test foobar(rand(1:1000)) == false
end
Though I’d precompute the random numbers and use the resulting index in the description string as well, since that’s what actually makes the test fail. Maybe like for i in rand(1:10000, 1000).
Thanks @Sukera. The problem is that I then end up with 10000 lines in my output:
Test Summary: | Pass Total
Sample test 1 | 1 1
Test Summary: | Pass Total
Sample test 2. | 1 1
Test Summary: | Pass Total
Sample test 3. | 1 1
…etc., until:
Test Summary: | Pass Total
Sample test 9999 | 1 1
Test Summary: | Pass Total
Sample test 10000 | 1 1
I would like run 10000 tests with the same name, BUT have the failure summaries tell me the i value that was used. Let’s take the following example:
Sample test: Test Failed at test_example.jl:9
Expression: foobar(rand(1:1000)) == false
Evaluated: true == false
(Stacktrace removed for brevity)
Test Summary: | Pass Fail Total
Sample test | 9993 7 10000
I’d like to be able to make the first line say
Sample test: Test Failed at test_example.jl:9 with i=2033
for example (spot the i=2033 at the end). That way my console isn’t flooded with text; it only reports on actual failures.
You can also nest @testset - that should only report failures. I do that for all my tests, with one big test set encompassing all smaller, more specific testsets.
This still prints all elements on a failure in the summary, but to be honest I’m not sure having such a large loop resulting in individual tests is a good idea. Another approach could be to keep a list of ids that fail and only @test whether that list is empty:
julia> @testset "failing ids" begin
a = [1,2,43]
@test isempty(a)
end
failing ids: Test Failed at REPL[5]:3
Expression: isempty(a)
Evaluated: isempty([1, 2, 43])
Stacktrace:
[..] # omitted for brevity
Test Summary: | Fail Total Time
failing ids | 1 1 0.3s
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
The upcoming 1.8 release will also allow @testset my_testset_function(), which may be helpful for your grouping as well.