Why Julia fails to infere types in this example? What is the cost of this? What are requirements for type inference?

Hi! I’ve opened a bug report (#54344) on github today which was immediately closed and I was sent here. I’ve read that for a good performance @code_warntype macro should produce an output without lines colored in red. For this purpose I should write code which is “type stable” and there is no “type uncertainty”. Consider the following code:

function foo(array, op, xs)
    function cc(:: Any)
        op(array)
    end

    return cc.(xs)
end

After evaluating this in the REPL I get the following:

julia> data = rand(Bool, (100, 100, 100));

julia> @code_warntype foo(data, x -> sum(2x), [2])
MethodInstance for foo(::Array{Bool, 3}, ::var"#2#3", ::Vector{Int64})
  from foo(array, op, ps1) @ Main REPL[1]:1
Arguments
  #self#::Core.Const(foo)
  array::Array{Bool, 3}
  op::Core.Const(var"#2#3"())
  ps1::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#2#3"}
Body::Any
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#2#3")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#2#3"})
│        (cc = %new(%4, array, op))
│   %6 = Base.broadcasted(cc, ps1)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, var"#cc#1"{Array{Bool, 3}, var"#2#3"}, Tuple{Vector{Int64}}}
│   %7 = Base.materialize(%6)::Any
└──      return %7

Body::Any is in red. I wonder why @code_warntype fails to give a narrower type like Vector{Int} here. What a penalty for this may be? What are the rules for writing a correct code? How does your type inference work?

If the rule is to write functions which return values of the same type for all possible arguments of the same type then this is definitely a bug. This is why I have reported it and I cannot understand why it was closed. If there are more rules, I would like to know an exhaustive list. This is really hard to tell if this is a bug or not because an end user like me has no idea how this inference works or what it proves unlike some well-known algorithms, be it Hindley-Milner like in Haskell or Kaplan-Ullman like in Common Lisp.

Update:* @code_warntype foo(data, sum, [3]) works fine, for example

What is the output of versioninfo()? I am unable to reproduce.

julia> @code_warntype foo(Array{Bool,3}(undef, 1, 2, 3), x -> sum(2x), [2])
MethodInstance for foo(::Array{Bool, 3}, ::var"#4#5", ::Vector{Int64})
  from foo(array, op, xs) @ Main REPL[5]:1
Arguments
  #self#::Core.Const(foo)
  array::Array{Bool, 3}
  op::Core.Const(var"#4#5"())
  xs::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#4#5"}
Body::Vector{Int64}
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#4#5")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#4#5"})
│        (cc = %new(%4, array, op))
│   %6 = Base.broadcasted(cc, xs)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, var"#cc#1"{Array{Bool, 3}, var"#4#5"}, Tuple{Vector{Int64}}}
│   %7 = Base.materialize(%6)::Vector{Int64}
└──      return %7


julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

Update, ok, I see the potential issue here. If I do a trial execution first, this infers.

julia> function foo(array, op, xs)
                  function cc(:: Any)
                      op(array)
                  end

                  return cc.(xs)
              end
foo (generic function with 1 method)

julia> foo(Array{Bool,3}(undef, 1, 2, 3), x -> sum(2x), [2])
1-element Vector{Int64}:
 0

julia> @code_warntype foo(Array{Bool,3}(undef, 1, 2, 3), x -> sum(2x), [2])
MethodInstance for foo(::Array{Bool, 3}, ::var"#4#5", ::Vector{Int64})
  from foo(array, op, xs) @ Main REPL[1]:1
Arguments
  #self#::Core.Const(foo)
  array::Array{Bool, 3}
  op::Core.Const(var"#4#5"())
  xs::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#4#5"}
Body::Vector{Int64}
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#4#5")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#4#5"})
│        (cc = %new(%4, array, op))
│   %6 = Base.broadcasted(cc, xs)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, var"#cc#1"{Array{Bool, 3}, var"#4#5"}, Tuple{Vector{Int64}}}
│   %7 = Base.materialize(%6)::Vector{Int64}
└──      return %7

If I do not do a trial execution, then I can reproduce.

julia> function foo(array, op, xs)
                  function cc(:: Any)
                      op(array)
                  end

                  return cc.(xs)
              end
foo (generic function with 1 method)

julia> @code_warntype foo(Array{Bool,3}(undef, 1, 2, 3), x -> sum(2x), [2])
MethodInstance for foo(::Array{Bool, 3}, ::var"#2#3", ::Vector{Int64})
  from foo(array, op, xs) @ Main REPL[1]:1
Arguments
  #self#::Core.Const(foo)
  array::Array{Bool, 3}
  op::Core.Const(var"#2#3"())
  xs::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#2#3"}
Body::Any
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#2#3")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#2#3"})
│        (cc = %new(%4, array, op))
│   %6 = Base.broadcasted(cc, xs)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, var"#cc#1"{Array{Bool, 3}, var"#2#3"}, Tuple{Vector{Int64}}}
│   %7 = Base.materialize(%6)::Any
└──      return %7

Sorry, @mkitti, I’ve completely forgotten about this!

julia> versioninfo()
Julia Version 1.10.3
Commit 0b4590a550 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: FreeBSD (x86_64-unknown-freebsd13.2)
  CPU: 16 × AMD Ryzen 7 5800X 8-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

Exactly, if I execute it, @code_warntype works as expected. Isn’t it weird?

This might be a particular instance of when Julia tries to avoid specializing.

See

https://docs.julialang.org/en/v1/manual/performance-tips/#Be-aware-of-when-Julia-avoids-specializing

2 Likes

I don’t know the reason, but in case someone can explain it to those who are not as knowledgeable: If instead of using broadcasting, you use the map function, @code_warntype returns the expected output.

I hope I’m not making a silly comparison; I know that using map is not always equivalent to a broadcast.

Summary
julia> @code_warntype foo(data, x -> sum(2x), [2])
MethodInstance for foo(::Array{Bool, 3}, ::var"#2#3", ::Vector{Int64})
  from foo(array, op, xs) @ Main REPL[2]:1
Arguments
  #self#::Core.Const(foo)
  array::Array{Bool, 3}
  op::Core.Const(var"#2#3"())
  xs::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#2#3"}
Body::Any
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#2#3")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#2#3"})
│        (cc = %new(%4, array, op))
│   %6 = Base.broadcasted(cc, xs)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, var"#cc#1"{Array{Bool, 3}, var"#2#3"}, Tuple{Vector{Int64}}}
│   %7 = Base.materialize(%6)::Any
└──      return %7

In a new session:

Summary
julia> function foo_map(array, op, xs)
           function cc(:: Any)
               op(array)
           end
           return map(cc,xs)
       end
foo_map (generic function with 1 method)

julia>

julia> @code_warntype foo_map(data, x -> sum(2x), [2])
MethodInstance for foo_map(::Array{Bool, 3}, ::var"#2#3", ::Vector{Int64})
  from foo_map(array, op, xs) @ Main REPL[2]:1
Arguments
  #self#::Core.Const(foo_map)
  array::Array{Bool, 3}
  op::Core.Const(var"#2#3"())
  xs::Vector{Int64}
Locals
  cc::var"#cc#1"{Array{Bool, 3}, var"#2#3"}
Body::Vector{Int64}
1 ─ %1 = Main.:(var"#cc#1")::Core.Const(var"#cc#1")
│   %2 = Core.typeof(array)::Core.Const(Array{Bool, 3})
│   %3 = Core.typeof(op)::Core.Const(var"#2#3")
│   %4 = Core.apply_type(%1, %2, %3)::Core.Const(var"#cc#1"{Array{Bool, 3}, var"#2#3"})
│        (cc = %new(%4, array, op))
│   %6 = Main.map(cc, xs)::Vector{Int64}
└──      return %6

Yes, it seems like a problem in broadcasting. I would like to use broadcasting in my particular case :frowning:

@mkitti I don’t know if the link you provided explains why this example works with map.

In general, Any is like Object in Java. It designates a variable that could potentially hold any value, so results in boxing.

Julias type inference is, due to being a semantically dynamic language, allowed to “give up” due to various (more or less known…) heuristics. In this case, it’s due to an inference failure in the broadcast materialization:

%7 = Base.materialize(%6)::Any Call inference reached maximally imprecise information.

(you can get this output if you inspect the call with Cthulhu.jl, through @descend foo(data, x -> sum(2x), [2]) and turning on compiler remarks)

Inferring Any is not (generally) considered a bug, because type inference is allowed to fail.

Here are some ressources that should be helpful:


I suspect that in your particular case, it’s the combination of having cc capture op and array as a closure while not using the sole argument to cc at all that leads to the inference failure.

The question may be asked in another way: how costly is it to work with boxed objects?

What can be done in this case to avoid type instability?

That’ll depend on what exactly you’re doing :person_shrugging: For cases where you’re basically doing just numerical work and you’d expect SIMD to occur, it’s catastrophic. Boxing with Any loses type information, so there’s no information about object layout. It’s basically a type check & dynamic dispatches everywhere.

On the other hand, for programs where you’re more-or-less doing dynamic dispatches anyway or the dynamicness is inherent to the problem at hand, it’s usually not a big deal at all. In fact, there’s quite a few places in Base too where you may want to explicitly opt out of specialized methods to reduce compilation time.

In general though, if the code would not have vectorized anyway and the instability is localized so far that there’s basically only one dynamic dispatch, it’s on the order of a few microseconds. Depending on the application, that can be a lot or nothing at all.

There’s a few general strategies, though how far they’re applicable to your actual code base can vary since the example you’ve presented seems kind of artificial. I generally tend to avoid closures and use map more often, so I’d just go for that. Is there a particular reason why you want to stick to broadcasting here?

1 Like

My real-life function computes three-point correlation for an array in a set of points like this:

function autocorr3_point(array  :: AbstractArray{<:Number, N},
                         shift2 :: NTuple{N, Int},
                         shift3 :: NTuple{N, Int}) where N

    rot1 = array
    rot2 = circshift(array, shift2)
    rot3 = circshift(array, shift3)
    return sum(rot1 .* rot2 .* rot3)
end

function s3(array, ps1, ps2) = autocorr3_point.(Ref(array), ps1, ps2)

autocorr3_point alone is not enough for me, because there are other functions which make other arrays out of array and later do the broadcasting.

Now try this:

julia> @code_warntype Shit.s3([1,2,3,4], [(0,)], [(0,)])
MethodInstance for Main.Shit.s3(::Vector{Int64}, ::Vector{Tuple{Int64}}, ::Vector{Tuple{Int64}})
  from s3(array, ps1, ps2) @ Main.Shit /usr/home/vasily/foo.jl:22
Arguments
  #self#::Core.Const(Main.Shit.s3)
  array::Vector{Int64}
  ps1::Vector{Tuple{Int64}}
  ps2::Vector{Tuple{Int64}}
Body::Any
1 ─ %1 = Main.Shit.autocorr3_point::Core.Const(Main.Shit.autocorr3_point)
│   %2 = Main.Shit.Ref(array)::Base.RefValue{Vector{Int64}}
│   %3 = Base.broadcasted(%1, %2, ps1, ps2)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(Main.Shit.autocorr3_point), Tuple{Base.RefValue{Vector{Int64}}, Vector{Tuple{Int64}}, Vector{Tuple{Int64}}}}
│   %4 = Base.materialize(%3)::Any
└──      return %4


julia> @code_warntype Shit.autocorr3_point([1,2,3,4], (0,), (0,))
MethodInstance for Main.Shit.autocorr3_point(::Vector{Int64}, ::Tuple{Int64}, ::Tuple{Int64})
  from autocorr3_point(array::AbstractArray{<:Number, N}, shift2::Tuple{Vararg{Int64, N}}, shift3::Tuple{Vararg{Int64, N}}) where N @ Main.Shit /usr/home/vasily/foo.jl:12
Static Parameters
  N = 1
Arguments
  #self#::Core.Const(Main.Shit.autocorr3_point)
  array::Vector{Int64}
  shift2::Tuple{Int64}
  shift3::Tuple{Int64}
Locals
  rot3::Vector{Int64}
  rot2::Vector{Int64}
  rot1::Vector{Int64}
Body::Int64
1 ─      (rot1 = array)
│        (rot2 = Main.Shit.circshift(array, shift2))
│        (rot3 = Main.Shit.circshift(array, shift3))
│   %4 = Main.Shit.:*::Core.Const(*)
│   %5 = Base.broadcasted(Main.Shit.:*, rot1, rot2)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(*), Tuple{Vector{Int64}, Vector{Int64}}}
│   %6 = Base.broadcasted(%4, %5, rot3)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(*), Tuple{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(*), Tuple{Vector{Int64}, Vector{Int64}}}, Vector{Int64}}}
│   %7 = Base.materialize(%6)::Vector{Int64}
│   %8 = Main.Shit.sum(%7)::Int64
└──      return %8

The result of autocorr3_point is Int64, but broadcasted autocorr3_point returns Any. What is happening?

The first version with the closure is another way to prevent broadcasting on array (I need to broadcast only on ps1 and ps2).

PS 1 (updated): I thought op is relevant here, but it is not. It’s enough that sum is applied to a result of a ternary operation of rot1, rot2 and rot3.

PS 2: A binary operation is enough here:

function autocorr2_point(array :: AbstractArray{<:Number, N},
                         shift :: NTuple{N, Int}) where N

    rot1 = array
    rot2 = circshift(array, shift)
    return sum(rot1 .* rot2)
end

s2(array, ps1) = autocorr2_point.(Ref(array), ps1)

Thank you for the reproducible example. Since you’re on 1.10.3, I suspect it’s an issue in that version (where I can reproduce the Any), as it infers just fine on the upcoming 1.11:

julia> @code_warntype s3([1,2,3,4], [(0,)], [(0,)])
MethodInstance for s3(::Vector{Int64}, ::Vector{Tuple{Int64}}, ::Vector{Tuple{Int64}})
  from s3(array, ps1, ps2) @ Main REPL[3]:1
Arguments
  #self#::Core.Const(s3)
  array::Vector{Int64}
  ps1::Vector{Tuple{Int64}}
  ps2::Vector{Tuple{Int64}}
Body::Vector{Int64}
1 ─ %1 = Main.autocorr3_point::Core.Const(autocorr3_point)
│   %2 = Main.Ref::Core.Const(Ref)
│   %3 = (%2)(array)::Base.RefValue{Vector{Int64}}
│   %4 = Base.broadcasted(%1, %3, ps1, ps2)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(autocorr3_point), Tuple{Base.RefValue{Vector{Int64}}, Vector{Tuple{Int64}}, Vector{Tuple{Int64}}}}
│   %5 = Base.materialize(%4)::Vector{Int64}
└──      return %5


julia> versioninfo()
Julia Version 1.11.0-alpha2
Commit 9dfd28ab751 (2024-03-18 20:35 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 24 × AMD Ryzen 9 7900X 12-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver4)
Threads: 1 default, 0 interactive, 1 GC (on 24 virtual cores)
Environment:
  JULIA_PKG_USE_CLI_GIT = true

Yes, I have reported this just in case on github after a while. @arhik was able to bisect the fix to aa42963. Thanks you all for helping!

But the first case is not fixed yet. I think I’ll rewrite my broadcasting from using a closure to using Refs.