# How to write a unit test that checks whether a function is type stable

**URL:** <https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587>\
**Category:** General Usage\
**Tags:** question, testing, inference, type-stability, feature-request\
**Created:** [October 4, 2023, 8:39pm UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587 "2023-10-04T20:39:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![CFBaptista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cfbaptista/32/9264_2.png) [@CFBaptista](https://discourse.julialang.org/u/CFBaptista)\
**Post date:** [October 4, 2023, 8:39pm UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/1 "2023-10-04T20:39:36Z")

</div>

I want to write unit tests that check whether a function is type stable.  
What is currently the most recommendable way to do this?

There is an [old post](https://discourse.julialang.org/t/testing-for-type-stability-regressions/10629/3) on this subject which mentions the [Traceur.jl](https://github.com/JunoLab/Traceur.jl) and [TraceCalls.jl](https://github.com/cstjean/TraceCalls.jl) packages for this purpose.  
The former has not been updated for two years and the latter is not supported beyond Julia 0.6.  
An [old PR](https://github.com/JuliaLang/julia/pull/23426) for adding a `@isinferred` macro that returns a boolean was never merged and appears abandoned.

`@inferred` gives an error if a function is type unstable.  
Unfortunately, it does not throw a specific exception: `ErrorException`.  
Checking for a generic exception is too wide and thus susceptible to false positives.  
Would it be desirable to modify `@inferred` such that it throws a suitable exception (e.g. `TypeInstabilityError` which does not exist for as far as I am aware) instead of `ErrorException`?  
That way the following would be possible:

```julia
using Test

function type_unstable_function(x, y)
    if x > y
        return x * y
    else
        return x / y
    end
end

@test_throws TypeInstabilityError @inferred type_unstable_function(1, 2)

```

**EDIT**  
I was made aware by @Benny that `@inferred` does throw an exception: `ErrorException`.  
I changed the above to correct my oversight.  
Also my test was missing `@inferred`, I have added it now.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 5, 2023, 1:04am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/2 "2023-10-05T01:04:44Z")

</div>

> [@CFBaptista](#):
>
> `@inferred` gives an error if a function is type unstable.  
> Unfortunately, it does not throw an exception.

I think I must be misunderstanding something here, because this is throwing an `ErrorException`:

```julia
julia> f(x) = x > 0 ? 1 : 1.0
f (generic function with 1 method)

julia> @inferred f(1)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}

```

In this example the small inferred Union isn’t actually a performance-killer so that might need to be distinguished in the proposed unit test.  
There is an extra layer where an internal call can be type-unstable, so I’m not sure if you also want this unit test to find that kind of instability. Example continuing from before:

```julia
julia> function g(x)::Float64 f(x) end
g (generic function with 1 method)

julia> @inferred g(1)
1.0

```

---

<div class="post-metadata">

**Author:** ![CFBaptista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cfbaptista/32/9264_2.png) [@CFBaptista](https://discourse.julialang.org/u/CFBaptista)\
**Post date:** [October 5, 2023, 2:36am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/3 "2023-10-05T02:36:43Z")

</div>

My bad, an exception is indeed thrown.  
However, an `ErrorException` is too generic.  
It might be thrown for reasons other than a type instability.  
My example is, however, too simple to show this.

The case you describe where an internal call is type unstable is more complex.  
Eventually I would also want to test for that.  
For now I would be satisfied if in cases where `@inferred` throws an exception due to type instability it would throw a more specific one.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 5, 2023, 2:53am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/4 "2023-10-05T02:53:37Z")

</div>

It would be good to open an issue on github about making the exception more specific.

But practice you can use `@inferred` to unit test if a function is type stable. (well really am checking if its inferable but lets assume that is close enough).  
The human readable error is enough (but I agree it could be better).  
Because either:

- your test suite because it is not inferable , and so you look at the error and see it saying as such.
- Or your test fails because the code is broken for some other reason and so you look to check and see that it doesn’t have that error message but rather something else you need to fix.
- Or it passes and you know it was inferrable and didn’t have another error.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 5, 2023, 7:56am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/5 "2023-10-05T07:56:09Z")

</div>

Note that `Test.@inferred` only tests the inference of the function _output_.  
If you need to test full type inferrability, use `JET.@test_opt` from [JET.jl](https://github.com/aviatesk/JET.jl)

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [October 5, 2023, 8:12am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/6 "2023-10-05T08:12:46Z")

</div>

Though it’s worth noting that for this example, JET thinks there’s nothing wrong with `type_unstable_function` from the OP.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [October 5, 2023, 8:19pm UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/7 "2023-10-05T20:19:06Z")

</div>

As in 2018, just check allocations. Type unstable code always allocates.

---

<div class="post-metadata">

**Author:** ![sschmidhuber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sschmidhuber/32/13349_2.png) [@sschmidhuber](https://discourse.julialang.org/u/sschmidhuber)\
**Post date:** [October 5, 2023, 9:32pm UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/8 "2023-10-05T21:32:00Z")

</div>

Hello @gdalle,  
I also found that test integration part of JET, but I’m not able to make it work.

If I run `@test_opt type_unstable_function(x, y)` it passes the test, seemingly independent of the values of x and y.  
Could you show an example how this function works, unfortunately there is only an example where the test passes in the documentation of JET, which is not helpful for my issue.

Thank you!

Cheers Stefan

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 6, 2023, 6:10am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/9 "2023-10-06T06:10:54Z")

</div>

Apparently @Mason confirmed that JET is not sensitive enough for this function, not sure why

---

<div class="post-metadata">

**Author:** ![CFBaptista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cfbaptista/32/9264_2.png) [@CFBaptista](https://discourse.julialang.org/u/CFBaptista)\
**Post date:** [October 6, 2023, 6:27am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/10 "2023-10-06T06:27:33Z")

</div>

Isn’t this susceptible to false positives?

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 6, 2023, 6:43am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/11 "2023-10-06T06:43:38Z")

</div>

I think JET looks for runtime dispatches specifically and thus ignores inferred small Unions that are optimized by [Union-splitting](https://julialang.org/blog/2018/08/union-splitting/) the code into static dispatch branches. The inferred return type of the example functions so far is `Union{Float64, Int64}`. I wonder if there’s an option to catch small Unions too, even if it’s not a performance hit by itself, it could lead to a runtime dispatch-instability in combination with other such methods e.g. `returns6types(returns2types(x), returns3types(y))`.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [October 6, 2023, 7:11am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/12 "2023-10-06T07:11:39Z")

</div>

Not really a source of false positives. If you don’t intentionally allocate in a function the allocations should be zero. If you do allocate, you know that.

---

<div class="post-metadata">

**Author:** ![CFBaptista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cfbaptista/32/9264_2.png) [@CFBaptista](https://discourse.julialang.org/u/CFBaptista)\
**Post date:** [October 6, 2023, 8:35am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/13 "2023-10-06T08:35:48Z")

</div>

Type instability involves allocation, but allocation not always involves type instability.  
I try to test for these behaviors separately, striving to test the behavior and not the implementation.  
When writing a unit test I would like not to make assumptions about the implementation (as far as possible of course).

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 6, 2023, 8:44am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/14 "2023-10-06T08:44:43Z")

</div>

Besides allocations having causes besides type instability so they can be false positives for an instability test, CFBaptista is considering the original example type-unstable because its inferred return type is not concrete, and it does not allocate

```julia
julia> @time type_unstable_function(1, 2)
  0.000001 seconds
0.5

```

So that’s also a false negative for instability by that standard.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [October 6, 2023, 8:58am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/15 "2023-10-06T08:58:47Z")

</div>

Small unions have no allocations because there are no boxed variables.

And even a for loop has small union type instability so testing it is also going to have its own false positives.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [October 6, 2023, 9:14am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/16 "2023-10-06T09:14:56Z")

</div>

> [@gdalle](#):
>
> Apparently @Mason confirmed that JET is not sensitive enough for this function, not sure why

yeah as @Benny mentioned, it’s basically just that JET rightfully realizes that this return type is quite unproblematic in normal circumstances.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [October 6, 2023, 9:25am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/17 "2023-10-06T09:25:17Z")

</div>

One additional potential source of false positives with allocation testing is that turning on code coverage in your test suite can cause allocations when there were none before (because it essentially adds IO calls all over your code).

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [October 6, 2023, 9:28am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/18 "2023-10-06T09:28:43Z")

</div>

My recommendation if you find type stability really really crucial for your usecases would be to use `@inferred` to ensure your return type is concrete and stable, and then to use `JET.@test_opt` to make sure the internals of your function aren’t encountering any problematic (i.e. not small union) type instabilities.

---

<div class="post-metadata">

**Author:** ![CFBaptista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cfbaptista/32/9264_2.png) [@CFBaptista](https://discourse.julialang.org/u/CFBaptista)\
**Post date:** [October 6, 2023, 9:36am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/19 "2023-10-06T09:36:53Z")

</div>

@Mason when you say to check whether the return type is concrete and stable do you mean something similar as was proposed by this [old PR](https://github.com/JuliaLang/julia/pull/23426)?

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [October 6, 2023, 9:46am UTC](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587/20 "2023-10-06T09:46:48Z")

</div>

🤷‍♂️

No, I meant just using `@inferred`, but if you don’t like `@inferred` (which is totally reasonable to not like), why not just use `Core.Compiler.return_type` directly? i.e.

```julia
using Test
using Core.Compiler: return_type
@test return_type(f, Tuple{T1, T2}) == T3

```

By the way though, precision of the type inference system is specifically something that is not a guaranteed API in julia, so if you write a test suite that will fail if the type inference becomes less precise in the future, your package may get delisted from the package eval system, which means that your package won’t be used to determined if a given change is a breaking change. This likely isn’t a big deal, but I just thought I’d point it out.

[Next page](https://discourse.julialang.org/t/how-to-write-a-unit-test-that-checks-whether-a-function-is-type-stable/104587.md?page=2)