Different result of `===` between `@runtest` and REPL

bthorne93 · January 17, 2024, 7:30pm

In an entirely clean package, the following unit test passes:

using Test 

@testset begin
    function build_matrix()
        A = zeros(2, 2)
        A[1, 2] = A[2, 1] = 1
        return A
    end
    
    A = build_matrix()
    B = build_matrix()
    @test A === B
    
end

Whereas, in the REPL, A===B returns false, which is what I would expect. What is going on here?

mkitti · January 17, 2024, 9:35pm

Hmm… I think it has to do with local scope, but I agree this is surprising.

Let me simplify the example.

julia> let
           function build_matrix()
               A = zeros(2,2)
               A[1,2] = A[2,1] = 1
               return A
           end
           A = build_matrix()
           B = build_matrix()
           A === B
       end
true

What is happening is that the binding A refers to A within the local scope of the let block. The second time we call build_matrix it reassigns A.

julia> let
           function build_matrix()
               A = zeros(2,2)
               @info "build_matrix: " pointer(A)
               A[1,2] = A[2,1] = 1
               return A
           end
           A = build_matrix()
           @info "A = build_matrix()" pointer(A)
           B = build_matrix()
           @info "B = build_matrix()" pointer(A) pointer(B)
           A === B
       end
┌ Info: build_matrix: 
└   pointer(A) = Ptr{Float64} @0x00007fd15df7a080
┌ Info: A = build_matrix()
└   pointer(A) = Ptr{Float64} @0x00007fd15df7a080
┌ Info: build_matrix: 
└   pointer(A) = Ptr{Float64} @0x00007fd15df7b340
┌ Info: B = build_matrix()
│   pointer(A) = Ptr{Float64} @0x00007fd15df7b340
└   pointer(B) = Ptr{Float64} @0x00007fd15df7b340
true

This applies to @testset since the macro creates a local scope using let.

julia> @macroexpand @testset begin
           function build_matrix()
               A = zeros(2, 2)
               A[1, 2] = A[2, 1] = 1
               return A
           end
           
           A = build_matrix()
           B = build_matrix()
           @test A === B
           println(A === B)
           
       end
quote
# ...
            let
                #= /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 =#
                begin
                    #= REPL[26]:2 =#
                    function build_matrix()
                        #= REPL[26]:2 =#
                        #= REPL[26]:3 =#
                        A = zeros(2, 2)
                        #= REPL[26]:4 =#
                        A[1, 2] = (A[2, 1] = 1)
                        #= REPL[26]:5 =#
                        return A
                    end
                    #= REPL[26]:8 =#
                    A = build_matrix()
                    #= REPL[26]:9 =#
                    B = build_matrix()
                    #= REPL[26]:10 =#

Conversely, if we define build_matrix in global scope, then we will get false.

julia> function build_matrix()
           A = zeros(2, 2)
           A[1, 2] = A[2, 1] = 1
           return A
       end
build_matrix (generic function with 1 method)

julia> @testset begin
           A = build_matrix()
           B = build_matrix()
           @test A === B
       end
test set: Test Failed at REPL[31]:4
  Expression: A === B
   Evaluated: [0.0 1.0; 1.0 0.0] === [0.0 1.0; 1.0 0.0]

raminammour · January 17, 2024, 9:37pm

Test is a red herring. Something is fishy here, I think it is worth filing an issue:


julia> g()=begin
           function build_matrix()
               A = zeros(2, 2)
               A[1, 2] = A[2, 1] = 1
               return A
           end
           
           A = build_matrix()
           B = build_matrix()
            A === B
           
       end
g (generic function with 1 method)

julia> h()=begin 
           function build_matrix()
               A = zeros(2, 2)
               A[1, 2] = A[2, 1] = 1
               return A
           end
           
           C = build_matrix()
           B = build_matrix()
            C === B
           
       end
h (generic function with 1 method)
julia> g(),h()
(true, false)

julia> @code_warntype optimize=:false h()
MethodInstance for h()
  from h() @ Main REPL[38]:1
Arguments
  #self#::Core.Const(h)
Locals
  B::Matrix{Float64}
  C::Matrix{Float64}
  build_matrix::var"#build_matrix#14"
Body::Bool
1 ─      (build_matrix = %new(Main.:(var"#build_matrix#14")))
│        (C = (build_matrix)())
│        (B = (build_matrix)())
│   %4 = (C === B)::Bool
└──      return %4


julia> @code_warntype optimize=:false g()
MethodInstance for g()
  from g() @ Main REPL[36]:1
Arguments
  #self#::Core.Const(g)
Locals
  B::Any
  A@_3::Core.Box
  build_matrix::var"#build_matrix#13"
  A@_5::Union{}
Body::Bool
1 ─       (A@_3 = Core.Box())
│         (build_matrix = %new(Main.:(var"#build_matrix#13"), A@_3))
│   %3  = (build_matrix)()::Any
│         Core.setfield!(A@_3, :contents, %3)
│         (B = (build_matrix)())
│   %6  = Core.isdefined(A@_3, :contents)::Bool
└──       goto #3 if not %6
2 ─       goto #4
3 ─       Core.NewvarNode(:(A@_5))
└──       A@_5
4 ┄ %11 = Core.getfield(A@_3, :contents)::Any
│   %12 = (%11 === B)::Bool
└──       return %12

Just renaming A to C yields expected results (the output of the @code_warntype with the same name A looks utterly confusing and confused)! This looks like a bug, on 1.9.0 at least.

Cheers!

abraemer · January 18, 2024, 9:17am

I also tested this on 1.8 and 1.10 and got the same results. It really seems that function does not introduce a proper local scope if inside a let and I don’t think that this can be intended.

Another “funny” consequence of this is that === appears non-commutative:

let
    function test()
        A = zeros(1)
        return A
    end
    A = test()
    A === test(), test() === A # gives (false, true)
end

abraemer · January 18, 2024, 9:28am

Actually I changed my mind and now think this is intended. From the manual on scopes (emphasis mine):

A new local scope is introduced by most code blocks (see above table for a complete list). If such a block is syntactically nested inside of another local scope, the scope it creates is nested inside of all the local scopes that it appears within, which are all ultimately nested inside of the global scope of the module in which the code is evaluated. **Variables in outer scopes are visible from any scope they contain […] unless there is a local variable with the same name that “shadows” the outer variable of the same name. This is true even if the outer local is declared after (in the sense of textually below) an inner block. When we say that a variable “exists” in a given scope, this means that a variable by that name exists in any of the scopes that the current scope is nested inside of, including the current one.
[…]
When x = <value> occurs in a local scope, Julia applies the following rules to decide what the expression means based on where the assignment expression occurs and what x already refers to at that location:

Existing local: If x is already a local variable, then the existing local x is assigned;

Hard scope: If x is not already a local variable and assignment occurs inside of any hard scope construct (i.e. within a let block, function or macro body, comprehension, or generator), a new local named x is created in the scope of the assignment;

I don’t like all consequences of this (like the example in this thread) but this is the way it is…

Edit: So the correct way to write snippets above is with using local inside the inner function:

mkitti:

julia> let
           function build_matrix()
               local A = zeros(2,2)
               A[1,2] = A[2,1] = 1
               return A
           end
           A = build_matrix()
           B = build_matrix()
           A === B
       end
false

Benny · January 18, 2024, 1:49pm

@testset’s local scoping is documented, though it doesn’t spell out how it’s different from the global scope like the scoping docs or this case.

The other more sinister possibility is that a local build_matrix can be reassigned, even by the function’s own call. The upside to local variables being reassigned by default within local scopes is this is how closures can work. Closures that reassign variables aren’t performant for now and are imo a bit strange, but there are situations where it’s called for. As long as you know the scoping rules and the @macroexpand isn’t too complex to check, closures won’t be much of a stumbling block, especially since in practice, you’d test globally scoped functions which can persist outside the test.

Something that can help is more descriptive names that differ inside and outside function blocks. Defaulting to quick but meaningless names is not too good a habit even if scoping rules help you out.

raminammour · January 18, 2024, 3:37pm

After some thought, it does look like this is intended, though unintuitive. So either:

Don’t use the same name of variables so that none of the “shadowing” behavior happens
If a function is meant to be defined in an outer scope like build_matrix, then just define it in the outer scope (as @mkitti did)

I guess this is what allows, sometimes desirable, behavior like:

julia> f1() = begin
       function bla()
       return a+1
       end

       a=3
       return a,bla()
       end
f1 (generic function with 1 method)

julia> f1()
(3, 4)

whereby defining bla() an outer scope would be an error.

Hope this helps…