# JET: are union incurred runtime dispatches really happening?

**URL:** <https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635>\
**Category:** Performance\
**Tags:** performance, jet\
**Created:** [July 4, 2024, 6:04pm UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635 "2024-07-04T18:04:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![filchristou](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/filchristou/32/26760_2.png) [@filchristou](https://discourse.julialang.org/u/filchristou)\
**Post date:** [July 4, 2024, 6:04pm UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/1 "2024-07-04T18:04:43Z")

</div>

I tend to think that the compiler is pretty smart and staff like union-splitting are handled gracefully under the hood.

However, I was currently using `JET.@code_opt` and I saw some dynamic dispatch where I thought there was none. Now, since JET is so dependent on the julia internals and compiler, I guess it doesn’t lie ?

The minimum working example is:

```julia
using JET

struct MyStruct
    alist::Union{Nothing, Vector{Int}}
end

function dosomething(m)
    return m.alist[1] + m.alist[2]
end

m = MyStruct([1,2,3]);

JET.@report_opt dosomething(m)

```

giving

```julia
═════ 2 possible errors found ═════
┌ dosomething(m::MyStruct) @ Main /u/home/wima/fchrstou/code/julia/jet/myjet.jl:6
│ runtime dispatch detected: (%1::Union{Nothing, Vector{Int64}})[1]::Int64
└────────────────────
┌ dosomething(m::MyStruct) @ Main /u/home/wima/fchrstou/code/julia/jet/myjet.jl:6
│ runtime dispatch detected: (%10::Union{Nothing, Vector{Int64}})[2]::Int64
└────────────────────

```

P.S.  
I know that the solution is to define

```julia
struct MyStruct2{T<:Union{Nothing, Vector{Int}}}
    alist::T
end

```

But I am wondering, on whether I can trust JET that julia indeed uses dynamic dispatch there ?

---

<div class="post-metadata">

**Author:** ![blegat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blegat/32/217090_2.png) [@blegat](https://discourse.julialang.org/u/blegat)\
**Post date:** [July 4, 2024, 7:34pm UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/2 "2024-07-04T19:34:02Z")

</div>

Try posting in [Performance - Julia Programming Language](https://discourse.julialang.org/c/usage/perf/37) instead, the field here is mathematical optimization, not optimization of code so you might have more answers there 😉

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [July 4, 2024, 8:01pm UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/3 "2024-07-04T20:01:50Z")

</div>

> [@filchristou](#):
>
> Now, since JET is so dependent on the julia internals and compiler, I guess it doesn’t lie ?

> [@filchristou](#):
>
> But I am wondering, on whether I can trust JET that julia indeed uses dynamic dispatch there ?

JET.jl does have bugs sometimes, but in this case it seems correct (checked with Cthulhu.jl). Not sure why union splitting isn’t helping here.

The weird thing is that even a `::Vector{Int}` type assertion can’t help here, it seems. I’ll try to put up a Julia bug report later.

---

<div class="post-metadata">

**Author:** ![aviatesk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aviatesk/32/7610_2.png) [@aviatesk](https://discourse.julialang.org/u/aviatesk)\
**Post date:** [July 5, 2024, 7:07am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/4 "2024-07-05T07:07:00Z")

</div>

The results of JET’s `report_opt` accurately reflect the behavior of the Julia compiler in most cases, so generally speaking you can trust them.  
Looking at the results of `@code_typed`, you will see several `:call`s to `Base.getindex(...)`:

```julia
CodeInfo(
1 ── %1 = Base.getfield(m, :alist)::Union{Nothing, Vector{Int64}}
│ %2 = (isa)(%1, Vector{Int64})::Bool
└─── goto #8 if not %2
2 ── %4 = π (%1, Vector{Int64})
│ %5 = $(Expr(:boundscheck, true))::Bool
└─── goto #6 if not %5
3 ── %7 = Base.sub_int(1, 1)::Int64
│ %8 = Base.bitcast(Base.UInt, %7)::UInt64
│ %9 = Base.getfield(%4, :size)::Tuple{Int64}
│ %10 = $(Expr(:boundscheck, true))::Bool
│ %11 = Base.getfield(%9, 1, %10)::Int64
│ %12 = Base.bitcast(Base.UInt, %11)::UInt64
│ %13 = Base.ult_int(%8, %12)::Bool
└─── goto #5 if not %13
4 ── goto #6
5 ── %16 = Core.tuple(1)::Tuple{Int64}
│ invoke Base.throw_boundserror(%4::Vector{Int64}, %16::Tuple{Int64})::Union{}
└─── unreachable
6 ┄─ %19 = Base.getfield(%4, :ref)::MemoryRef{Int64}
│ %20 = Base.memoryrefnew(%19, 1, false)::MemoryRef{Int64}
│ %21 = Base.memoryrefget(%20, :not_atomic, false)::Int64
└─── goto #7
7 ── goto #9
8 ── %24 = Base.getindex(%1, 1)::Int64
└─── goto #9
9 ┄─ %26 = φ (#7 => %21, #8 => %24)::Int64
│ %27 = Base.getfield(m, :alist)::Union{Nothing, Vector{Int64}}
│ %28 = (isa)(%27, Vector{Int64})::Bool
└─── goto #16 if not %28
10 ─ %30 = π (%27, Vector{Int64})
│ %31 = $(Expr(:boundscheck, true))::Bool
└─── goto #14 if not %31
11 ─ %33 = Base.sub_int(2, 1)::Int64
│ %34 = Base.bitcast(Base.UInt, %33)::UInt64
│ %35 = Base.getfield(%30, :size)::Tuple{Int64}
│ %36 = $(Expr(:boundscheck, true))::Bool
│ %37 = Base.getfield(%35, 1, %36)::Int64
│ %38 = Base.bitcast(Base.UInt, %37)::UInt64
│ %39 = Base.ult_int(%34, %38)::Bool
└─── goto #13 if not %39
12 ─ goto #14
13 ─ %42 = Core.tuple(2)::Tuple{Int64}
│ invoke Base.throw_boundserror(%30::Vector{Int64}, %42::Tuple{Int64})::Union{}
└─── unreachable
14 ┄ %45 = Base.getfield(%30, :ref)::MemoryRef{Int64}
│ %46 = Base.memoryrefnew(%45, 2, false)::MemoryRef{Int64}
│ %47 = Base.memoryrefget(%46, :not_atomic, false)::Int64
└─── goto #15
15 ─ goto #17
16 ─ %50 = Base.getindex(%27, 2)::Int64
└─── goto #17
17 ┄ %52 = φ (#15 => %47, #16 => %50)::Int64
│ %53 = Base.add_int(%26, %52)::Int64
└─── return %53
) => Int64

```

These correspond to the union split branches where `m.alist` is `nothing`. In such cases, a `MethodError` is raised, and since such “must-throw” case is not optimized, it results in dynamic dispatch. However, [Inline statically known method errors. by gbaraldi · Pull Request #54972 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/54972) might change this behavior.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 5, 2024, 7:53am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/5 "2024-07-05T07:53:24Z")

</div>

> [@aviatesk](#):
>
> ```julia
> │ %2 = (isa)(%1, Vector{Int64})::Bool
> └─── goto #8 if not %2
> ...
> 8 ── %24 = Base.getindex(%1, 1)::Int64
> └─── goto #9
> 
> ```

What’s going on here, It seems like it says if `nothing` is there, indexing it is inferred as `Int64`. Why would it infer that when it would throw an error here?

---

<div class="post-metadata">

**Author:** ![filchristou](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/filchristou/32/26760_2.png) [@filchristou](https://discourse.julialang.org/u/filchristou)\
**Post date:** [July 5, 2024, 7:56am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/6 "2024-07-05T07:56:30Z")

</div>

> [@nsajko](#):
>
> but in this case it seems correct (checked with [Cthulhu.jl](https://juliahub.com/ui/Packages/General/Cthulhu))

Hey @nsajko . How exactly did you find out that with Cthulhu ?  
I am only getting the following

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/1/c1600a5b476821ac25eda33de6aefcaa95aedd8a.png)  
, from which I cannot deduct dynamic dispatch is happening

---

<div class="post-metadata">

**Author:** ![filchristou](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/filchristou/32/26760_2.png) [@filchristou](https://discourse.julialang.org/u/filchristou)\
**Post date:** [July 5, 2024, 8:11am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/7 "2024-07-05T08:11:18Z")

</div>

That is pretty huge. So code like the following which I thought was dynamic-dispatch-free is actually not:

```julia

struct MyStruct
    el::Union{Nothing, Int}
end

function dosomething(m)
    # or if !(m.el isa Nothing)
    # or if !(m.el === nothing)
    if !isnothing(m.el) 
        return m.el + 1
    end
end

julia> m = MyStruct(1);

julia> JET.@report_opt dosomething(m)
═════ 1 possible error found ═════
┌ dosomething(m::MyStruct) @ Main /u/home/wima/fchrstou/code/julia/jet/myjet.jl:9
│ runtime dispatch detected: (%13::Union{Nothing, Int64} + 1)::Int64
└────────────────────

```

my best hopes for [Inline statically known method errors. by gbaraldi · Pull Request #54972 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/54972) 🙏

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 5, 2024, 8:32am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/8 "2024-07-05T08:32:31Z")

</div>

If I’m reading aviatesk’s comment correctly, the dynamic dispatch occurs when indexing `nothing` throws an error, not anywhere in the vector branch. That lines up with non-erroring calls having zero allocations and the overall method being inferrable. Something similar happens with FunctionWrappers.jl; it is often used to eliminate dynamic dispatch, but JET.jl will still report one in a precompile-time initialization branch that usually never runs.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 5, 2024, 8:41am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/9 "2024-07-05T08:41:58Z")

</div>

> [@filchristou](#):
>
> That is pretty huge. So code like the following which I thought was dynamic-dispatch-free is actually not:

Julia is (currently, IIRC there were plans to improve this) pretty bad at preserving these kinds of invariants on field access across different calls to `getproperty`. There won’t be dynamic dispatch if you write it like this:

```julia
julia> function dosomething(m)
           el = m.el
           if !isnothing(el) 
               return el + 1
           end
       end
dosomething (generic function with 1 method)

julia> JET.@report_opt dosomething(m)
No errors detected

```

This is because keeping that `!isnothing` information around is correctly tracked for the same variable there. The erroring path being the culprit is IMO a red herring.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [July 5, 2024, 10:23am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/10 "2024-07-05T10:23:32Z")

</div>

> [@filchristou](#):
>
> How exactly did you find out that with Cthulhu ?

Press `T`, for `[T]yped code`. EDIT: I’m not sure any more, though, does an entry marked as `runtime` mean run time dispatch, or does it mean something else…

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [July 5, 2024, 11:01am UTC](https://discourse.julialang.org/t/jet-are-union-incurred-runtime-dispatches-really-happening/116635/11 "2024-07-05T11:01:46Z")

</div>

> [@nsajko](#):
>
> The weird thing is that even a `::Vector{Int}` type assertion can’t help here, it seems. I’ll try to put up a Julia bug report later.

Fresh REPL:

```julia-repl
julia> using JET

julia> struct MyStruct
           alist::Union{Nothing,Vector{Int}}
       end

julia> function f(m)
           a = m.alist
           a::Vector{Int}
           a[1]
       end
f (generic function with 1 method)

julia> report_opt(f, Tuple{MyStruct})
═════ 1 possible error found ═════
┌ f(m::MyStruct) @ Main ./REPL[3]:4
│ runtime dispatch detected: (%1::Union{Nothing, Vector{Int64}})[1]::Int64
└────────────────────

julia> function g(m)
           a = m.alist
           (a::Vector{Int})[1]
       end
g (generic function with 1 method)

julia> report_opt(g, Tuple{MyStruct})
No errors detected

```

So the type assertion prevents run time dispatch in one simple case, but not in the other? This was with Julia v1.11, because JET.jl doesn’t work for nightly Julia.

@aviatesk how can I check if this issue persists on nightly Julia.

EDIT: issue here:

> <https://github.com/JuliaLang/julia/issues/55043>
>
> Reproducer:
> 
> \`\`\`julia
> using JET
> 
> struct MyStruct
> alist::Union{Nothing,…Vector{Int}}
> end
> 
> function f(m)
> a = m.alist
> a::Vector{Int}
> a\[1\]   
> end
> 
> 
> report\_opt(f, Tuple{MyStruct}) # run time dispatch
> \`\`\`
> 
> This is with Julia v1.11, as JET doesn't work on nightly Julia:
> 
> \`\`\`julia-repl
> julia\> versioninfo()
> Julia Version 1.11.0-beta2
> Commit edb3c92d6a6 (2024-05-29 09:37 UTC)
> Build Info:
> Official https://julialang.org/ release
> Platform Info:
> OS: Linux (x86\_64-linux-gnu)
> CPU: 8 × AMD Ryzen 3 5300U with Radeon Graphics
> WORD\_SIZE: 64
> LLVM: libLLVM-16.0.6 (ORCJIT, znver2)
> Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
> \`\`\`
> 
> If this:
> 
> \`\`\`julia
> a::Vector{Int}
> a\[1\]
> \`\`\`
> 
> ... is changed to:
> 
> \`\`\`julia
> (a::Vector{Int})\[1\]
> \`\`\`
> 
> ... then there's no run time dispatch!