I recently went on a bit of a rampage updating packages in the JuliaImages ecosystem, and on Julia 1.10 I noticed quite a few packages that had started failing their tests, with the log reports showing a pattern like this:
Core: Error During Test at file:line
Expression evaluated to non-Boolean
Expression: #= file:line =# @inferred foo(args...)
Value: [1 2 3]
That Expression evaluated to a non-Boolean coupled with @inferred is the fingerprint of this category of error.
This is almost surely a bug in the package’s test suite, not a bug in Julia. It comes from a test like this:
@test_broken @inferred foo(args...)
The intent here is to say “I’d like inference to succeed for this task, but it doesn’t currently, so I’m marking this as something to keep track of.” The @test_broken is acting like a try/catch on the error (one that prints a nice summary), preventing the failure in inference to cause failure in the aggregate test suite.
But now consider what happens when inference improves, as indeed it has in Julia 1.10, and the @inferred no longer errors. Now the @test_broken receives the return value of foo(args...), which because of the @test_broken means it will pass only if it returns false. If it returns something besides a Bool, you get the error above.
How to fix? The test should be written @test_broken @inferred(foo(args...)) == outputvalue. When inference starts succeeding, this too will fail, but it will fail “in the expected way” and a warning issued that the test should be changed to passing. Of course, you probably want to first make sure you’ve written the test correctly by checking it without the @inferred.