I want to write unit tests that check whether a function is type stable.
What is currently the most recommendable way to do this?
There is an old post on this subject which mentions the Traceur.jl and TraceCalls.jl packages for this purpose.
The former has not been updated for two years and the latter is not supported beyond Julia 0.6.
An old PR for adding a @isinferred macro that returns a boolean was never merged and appears abandoned.
@inferred gives an error if a function is type unstable.
Unfortunately, it does not throw a specific exception: ErrorException.
Checking for a generic exception is too wide and thus susceptible to false positives.
Would it be desirable to modify @inferred such that it throws a suitable exception (e.g. TypeInstabilityError which does not exist for as far as I am aware) instead of ErrorException?
That way the following would be possible:
using Test
function type_unstable_function(x, y)
if x > y
return x * y
else
return x / y
end
end
@test_throws TypeInstabilityError @inferred type_unstable_function(1, 2)
EDIT
I was made aware by @Benny that @inferred does throw an exception: ErrorException.
I changed the above to correct my oversight.
Also my test was missing @inferred, I have added it now.
I think I must be misunderstanding something here, because this is throwing an ErrorException:
julia> f(x) = x > 0 ? 1 : 1.0
f (generic function with 1 method)
julia> @inferred f(1)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}
In this example the small inferred Union isn’t actually a performance-killer so that might need to be distinguished in the proposed unit test.
There is an extra layer where an internal call can be type-unstable, so I’m not sure if you also want this unit test to find that kind of instability. Example continuing from before:
julia> function g(x)::Float64 f(x) end
g (generic function with 1 method)
julia> @inferred g(1)
1.0
My bad, an exception is indeed thrown.
However, an ErrorException is too generic.
It might be thrown for reasons other than a type instability.
My example is, however, too simple to show this.
The case you describe where an internal call is type unstable is more complex.
Eventually I would also want to test for that.
For now I would be satisfied if in cases where @inferred throws an exception due to type instability it would throw a more specific one.
It would be good to open an issue on github about making the exception more specific.
But practice you can use @inferred to unit test if a function is type stable. (well really am checking if its inferable but lets assume that is close enough).
The human readable error is enough (but I agree it could be better).
Because either:
your test suite because it is not inferable , and so you look at the error and see it saying as such.
Or your test fails because the code is broken for some other reason and so you look to check and see that it doesn’t have that error message but rather something else you need to fix.
Or it passes and you know it was inferrable and didn’t have another error.
Hello @gdalle,
I also found that test integration part of JET, but I’m not able to make it work.
If I run @test_opt type_unstable_function(x, y) it passes the test, seemingly independent of the values of x and y.
Could you show an example how this function works, unfortunately there is only an example where the test passes in the documentation of JET, which is not helpful for my issue.
I think JET looks for runtime dispatches specifically and thus ignores inferred small Unions that are optimized by Union-splitting the code into static dispatch branches. The inferred return type of the example functions so far is Union{Float64, Int64}. I wonder if there’s an option to catch small Unions too, even if it’s not a performance hit by itself, it could lead to a runtime dispatch-instability in combination with other such methods e.g. returns6types(returns2types(x), returns3types(y)).
Not really a source of false positives. If you don’t intentionally allocate in a function the allocations should be zero. If you do allocate, you know that.
Type instability involves allocation, but allocation not always involves type instability.
I try to test for these behaviors separately, striving to test the behavior and not the implementation.
When writing a unit test I would like not to make assumptions about the implementation (as far as possible of course).
Besides allocations having causes besides type instability so they can be false positives for an instability test, CFBaptista is considering the original example type-unstable because its inferred return type is not concrete, and it does not allocate
One additional potential source of false positives with allocation testing is that turning on code coverage in your test suite can cause allocations when there were none before (because it essentially adds IO calls all over your code).
My recommendation if you find type stability really really crucial for your usecases would be to use @inferred to ensure your return type is concrete and stable, and then to use JET.@test_opt to make sure the internals of your function aren’t encountering any problematic (i.e. not small union) type instabilities.
No, I meant just using @inferred, but if you don’t like @inferred (which is totally reasonable to not like), why not just use Core.Compiler.return_type directly? i.e.
using Test
using Core.Compiler: return_type
@test return_type(f, Tuple{T1, T2}) == T3
By the way though, precision of the type inference system is specifically something that is not a guaranteed API in julia, so if you write a test suite that will fail if the type inference becomes less precise in the future, your package may get delisted from the package eval system, which means that your package won’t be used to determined if a given change is a breaking change. This likely isn’t a big deal, but I just thought I’d point it out.
