# Testing type stability across Julia and JET versions

**URL:** <https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289>\
**Category:** General Usage\
**Tags:** testing, type-stability, jet\
**Created:** [June 27, 2024, 5:34am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289 "2024-06-27T05:34:53Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [June 27, 2024, 5:34am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/1 "2024-06-27T05:34:53Z")

</div>

In my package DifferentiationInterface.jl, the test suite includes type-stability checks for some differentiation operators. I perform these checks with JET.jl and the `@test_opt` macro, which actually verifies a stronger property called type-groundedness (according to the terminology in [this paper](https://arxiv.org/abs/2109.01950)). In other words, it’s not just about the output of the function being `@inferred`, but everything else that happens inside too.

My problem is that these tests are brittle: Julia doesn’t guarantee its inference behavior as part of the API, and JET is evolving fast. As a result, the type-stability tests often [break without me changing anything](https://github.com/gdalle/DifferentiationInterface.jl/actions/runs/9686665547/job/26729616486), with errors like

```julia
failed to optimize due to recursion

```

In addition, I often observe that the issue lies in some printing or string function that I don’t really care about (and didn’t even know would be called).

Is there a recommended way to make such tests more robust? Or less dependent on internals? Ping @aviatesk

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [June 27, 2024, 7:26am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/2 "2024-06-27T07:26:12Z")

</div>

Regarding extraneous issues reported:  
One approach is to specify a target module, e.g.

```julia
@report_opt target_modules=(@ __MODULE__ ,) compute(30)

```

which would only report issues in that module.  
Also,

> There is also [`function_filter`](https://aviatesk.github.io/JET.jl/dev/optanalysis/#optanalysis-config), which can ignore specific function calls.

In general, the [Optimization Analysis · JET.jl](https://aviatesk.github.io/JET.jl/dev/optanalysis/#optanalysis-config) page suggests some useful ideas.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [June 27, 2024, 7:27am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/3 "2024-06-27T07:27:42Z")

</div>

My hesitation with these tools is that I have two competing goals:

- Ensuring that my package itself doesn’t introduce type instabilities
- Testing that the code as a whole is type-stable, including function calls from other packages

Ignoring modules or functions is good for the first goal but bad for the second. Perhaps I should separate them?

---

<div class="post-metadata">

**Author:** ![aviatesk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aviatesk/32/7610_2.png) [@aviatesk](https://discourse.julialang.org/u/aviatesk)\
**Post date:** [June 27, 2024, 4:23pm UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/4 "2024-06-27T16:23:44Z")

</div>

Regarding this report in particular, it is expected to be fixed in the latest v1.11 ([1.11: improve type stability of `_unsafe_take!(::IOBuffer)` by aviatesk · Pull Request #54942 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/54942)).  
While I completely agree that such breakages are troublesome, there is also an aspect that such breakages are inevitable due to JET’s design. As @jishnub mentioned, using `function_filter` or `target_modules` is one way, but these do the opposite of @gdalle’s second objective. So this is another instance of common tradeoffs of analysis accuracy and false positives.  
In cases like this, I have usually fixed the issue on the Base side.  
That said, it’s true that the performance of logging-related code is rarely an actual problem, so it might be reasonable to make JET automatically ignore such code. JET is not a sound analyzer to begin with, and although this would mean adding a new analysis option, having too many options has already been an ongoing issue.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [June 27, 2024, 5:49pm UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/5 "2024-06-27T17:49:17Z")

</div>

Thanks for taking the time to answer!  
I think the best solution in my case would be to parametrize my function filter or ignored modules, depending on which of my goals I’m pursuing at the moment. I was worried that the macro arguments wouldn’t accept more complicated constructs than function names, but this seems to work:

```julia
julia> using JET

julia> function f(x)
       a = []
       push!(a, x)
       return sum(a)
       end
f (generic function with 1 method)

julia> @test_opt function_filter=Returns(false) f(1.0)
Test Passed

```

> [@aviatesk](#):
>
> JET is not a sound analyzer to begin with

What do you mean by that? If I’m not mistaken, JET is the only way to test type-groundedness in a programmatic way, without manual inspecting of the Cthulhu.jl results?

---

<div class="post-metadata">

**Author:** ![aviatesk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aviatesk/32/7610_2.png) [@aviatesk](https://discourse.julialang.org/u/aviatesk)\
**Post date:** [June 28, 2024, 7:03am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/6 "2024-06-28T07:03:29Z")

</div>

> [@gdalle](#):
>
> I was worried that the macro arguments wouldn’t accept more complicated constructs than function names, but this seems to work:

Yeah, `function_filter` takes on function objects rather than their names, so you can implement a filter that does a nuanced work.

> [@gdalle](#):
>
> What do you mean by that?

Ah, “sound” is a term used in static analysis and statistics, meaning something like “reliable”.  
What this actually means is that there are no false negatives; a sound analysis is guaranteed to detect all possible errors or issues, while it tends to often produce unnecessary warnings (false positives). In contrast, JET is not sound and prioritizes reducing false positives to make the tool more user-friendly rather than strictly eliminating false negatives. For example, ignoring dynamic dispatch in logging code or printing would typically result in false negatives for an optimization checker (`report_opt`), but from the user’s perspective in this case, it should be considered a false positive.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 28, 2024, 9:26am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/7 "2024-06-28T09:26:37Z")

</div>

> [@gdalle](#):
>
> What do you mean by that? If I’m not mistaken, JET

Ref

> **[Chapter 1 — Program Analysis](https://aviatesk.github.io/posts/introduction-to-static-analysis/chapter1/)**
>
> A note on

---

<div class="post-metadata">

**Author:** ![albertomercurio](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albertomercurio/32/27051_2.png) [@albertomercurio](https://discourse.julialang.org/u/albertomercurio)\
**Post date:** [December 22, 2024, 1:24am UTC](https://discourse.julialang.org/t/testing-type-stability-across-julia-and-jet-versions/116289/8 "2024-12-22T01:24:36Z")

</div>

Hi,

I’m facing the same issue. I get many errors related to `show` and similar function. I have to run something like

```julia
# Related to the `check_error` function inside the `integrator` interface
const sci_ml_integrator_functions = (Base.modulesof!, Base.show, Base.show_at_namedtuple, Base.show_typealias, Base._show_type, Base.isvisible, Base.eltype)

# Related to FFTW.jl
const fftw_functions = (QuantumToolbox.unsafe_destroy_plan, )

function_filter3(@nospecialize f) = f ∉ (sci_ml_integrator_functions..., fftw_functions...)

report = @report_opt function_filter=function_filter3 run_all_functions()

```

to avoid those issues. Is there a way to fix it? I think it is related to the `@warn` calls.

Moreover, my package uses OrdinaryDiffEq.jl, and, when I call `report_call`, I get errors like

```julia
BuiltinErrorReport(type NamedTuple has no field sensealg: Base.getfield(t::@NamedTuple{…}, i::Symbol))

```

but the code is something like

```julia
function solve(prob::AbstractDEProblem, args...; sensealg = nothing,
        u0 = nothing, p = nothing, wrap = Val(true), kwargs...)
    if sensealg === nothing && haskey(prob.kwargs, :sensealg)
        sensealg = prob.kwargs[:sensealg]
    end

...

```

where `prob.kwargs` should be defined at compile time. I really don’t understand.