Unit test for constant folding success

Here’s a proposal for how to test for successful constant folding:

using Test

function test_constant_folding(f, r)
    vec = code_typed(f, Tuple{})
    p = only(vec)
    code_info = first(p)
    code = try
        code_info.code
    catch
        nothing
    end
    @test repr(only(code)) == repr(:(return $r)) skip=(code isa Nothing)
end

# some example functions to test for foldability
function f(value = 9)
    it_fil = Iterators.filter((_ -> true), value)
    it_map = Iterators.map((x -> x + 1), it_fil)
    v = collect(it_map)
    only(v)
end
function g(collection = (2, 6))
    it_fil = Iterators.filter((_ -> true), collection)
    it_map = Iterators.map((x -> x + 1), it_fil)
    v = collect(it_map)
    sum(v)
end

@testset "constant folding" begin
    test_constant_folding((() -> 7), 7)
    test_constant_folding(f, 10)
    test_constant_folding(g, 10)
end

In the REPL:

julia> @testset "constant folding" begin
           test_constant_folding((() -> 7), 7)
           test_constant_folding(f, 10)
           test_constant_folding(g, 10)
       end
constant folding: Error During Test at REPL[2]:11
  Test threw exception
  Expression: repr(only(code)) == repr(:(return $r))
  ArgumentError: Collection has multiple elements, must contain exactly 1 element
  Stacktrace:
   [1] _only
     @ ./iterators.jl:1545 [inlined]
   [2] only(x::Vector{Any})
     @ Base.Iterators ./iterators.jl:1536
   [3] macro expansion
     @ ~/tmp/jl/jl/nightly_normal/julia-3d85309e80/share/julia/stdlib/v1.12/Test/src/Test.jl:676 [inlined]
   [4] test_constant_folding(f::Function, r::Int64)
     @ Main ./REPL[2]:11
constant folding: Error During Test at REPL[2]:11
  Test threw exception
  Expression: repr(only(code)) == repr(:(return $r))
  ArgumentError: Collection has multiple elements, must contain exactly 1 element
  Stacktrace:
   [1] _only
     @ ./iterators.jl:1545 [inlined]
   [2] only(x::Vector{Any})
     @ Base.Iterators ./iterators.jl:1536
   [3] macro expansion
     @ ~/tmp/jl/jl/nightly_normal/julia-3d85309e80/share/julia/stdlib/v1.12/Test/src/Test.jl:676 [inlined]
   [4] test_constant_folding(f::Function, r::Int64)
     @ Main ./REPL[2]:11
Test Summary:    | Pass  Error  Total  Time
constant folding |    1      2      3  2.1s
RNG of the outermost testset: Random.Xoshiro(0x2bc1148ebb7c1f60, 0x8ecccae6baa3e4fe, 0xc9fc8bc1405b0303, 0xd75d65d894272b1b, 0xada3f8f53a410b26)
ERROR: Some tests did not pass: 1 passed, 0 failed, 2 errored, 0 broken.

I think this test is supposed to work as expected as long as CodeInfo has an iterator of instructions in its :code field. When the property access fails, the test should be skipped gracefully, though.

I wonder if anyone has comments, in particular regarding reliability of the test. Are there any unexpected ways this could fail? cc @Sukera

Regarding possible alternatives, a weaker test would be to check, e.g., with Base.infer_effects or InteractiveUtils.@infer_effects (neither is public :sweat_smile:), whether the call infers as :foldable. However that doesn’t guarantee that it’ll actually be constant folded, AFAIK. Also, there’s no public API to check whether an Effects is :foldable, even though we have the non-public Core.Compiler.is_foldable.

Yes, this can fail in CI due to code coverage metadata. I’ve done a similar implementation in RequiredInterfaces.jl and had to add a bunch of almost arbitrary additional guards just for handling coverage correctly. It’s a very brittle piece of code that breaks often, and I’m not even trying to do this for arbitrary code, so I imagine it would be even worse for your usecase.

1 Like

Ah, yes, I know about that. We sort of work around that in FixedSizeArrays.jl by having a separate production build in CI, without --check-bounds=yes or code coverage, so it’s supposed to have good effects.

I was looking more for interfaces the test depends on, but that might break in future Julia versions.

There are none that I know of. As far as I know, effects are the only correct way to do this and they are not public/reliable API yet but rather an internal implementation detail of the compiler. The only other option I can think of is checking whether the compiler internally is able to infer the return “type” as a Core.Const, but this is also highly internal, so not a solution for you.

What’s wrong with my approach above, based on code_typed? code_typed is public, for what that’s worth. Its interface isn’t completely specified, so my idea is to skip the test if anything unexpected happens.

It relies on repr being invariant under quote interpolation, which may not be the case. For simple cases like integers and floats this works, because these objects are literals & placed into the IR directly, but for more complicated expressions this is no longer the case:

julia> struct Foo
           a::Int
       end

julia> test_constant_folding(() -> Foo(1), Foo(1))
Test Failed at REPL[9]:10
  Expression: repr(only(code)) == repr(:(return $r))
   Evaluated: ":(return \$(QuoteNode(Foo(1))))" == ":(return Foo(1))"

ERROR: There was an error during testing

Even without repr your property doesn’t hold, because interpolation is not in general the same as when a value is constructed by the compiler:

#=
           dump(only(code))
           cmp = :(return $r)
           dump(cmp)
=#
Core.ReturnNode
  val: QuoteNode
    value: Foo
      a: Int64 1
Expr
  head: Symbol return
  args: Array{Any}((1,))
    1: Foo
      a: Int64 1

That’s why I’m saying that the “most blessed” way of doing this is through the constant folding effects/type inference.

1 Like

True, thank you. I think the above code may still be useful for some test suites, though, as long as the return type in question works for repr, e.g., integers and floats, as you say.