Language restriction to solve 15276?

Internals & Design
41

See

42 
local x::Int 
x::Int = 1

https://docs.julialang.org/en/stable/manual/types/#Type-Declarations-1

Last time I checked that did help with the 15276 cases, but if it doesn’t with any of them then still that should be the appropriate syntax since it already exists in the language for declaring a constant type in a local scope.

Edit: I think there may be some confusion in that comment because of how it was written? @Stephen_Vavasis wrote a type assertion instead of a declaration.

1 Like
43

typeconstant x = 1 would be a bit easier cause you don’t have to keep track of what type x is, tho?

Edit:

Maybe a @typeconstant macro would do which is sugar for

x = 1 => local x::typeof(1) = 1?

using MacroTools: postwalk, @capture

macro typeconstant(e)
    esc(postwalk(e) do e
        if @capture e a_ = b_
            :(local $a::$typeof($b) = $b)
        else
            e
        end
    end)
end
44

Just to be clear: I’m suggesting that the declaration that a captured variable has a definite type should be inside the closure rather than in the enclosing scope. If I understand Matt Bauman’s description of how closures are parsed, it seems that the parser needs to make a decision about captured variables with only a limited understanding of the outer scope context.

45

Right. Putting a “type declaration” on a local variable is a purely syntactic form:

local x::T  # in a local declaration
x::T = 10   # as the left-hand side of an assignment

You’ve now asserted that x is a T (whatever that is bound to). So when you look at:

julia> Meta.@lower function f()
           if true
           end
           r::Int = 1
           cb = ()->r
           cb()
       end

You’ll see that Julia still creates a Box for r, but this time it also adds converts and typeasserts around assignments and accesses. This gives inference enough information to restore type stability. It doesn’t completely remove the Box, since assignments to r still need to be reflected in the callback and the parser wasn’t able to prove that r doesn’t change.

46

Has the radical solution of making an implicit let-block default been considered? This would be equivalent to changing the scoping rules for closures: They would get the same scoping rules as functions, i.e. always close over values and never over bindings. In my experience, this is what I want 99% of the time.

Then, one could not change the outer scope’s bindings anymore. This is irrelevant for mutating the captures, but would demand that user always construct a mutable container for captures if so desired, e.g. replacing an Int by Ref{Int}, or replacing an x::Array by Ref{typeof(x)}(x) (for some weird reason Ref(x::Array) behaves rather counter-intuitively on 0.62).

With proper support (e.g. helpful types and macros), I think this might be a good thing and much more intuitive for users: It always annoys me that the following gives different results in the top-level scope and inside a function:

function foo()
       r = 0;
       f1=()->r;
       f2=(x)-> (r=x);
       f2(1)
       @show f1();
       f1,f2
end
foo();
6 Likes
47

For the remaining 1 %, maybe one could use the outer keyword, same as in 0.7 for loops.

f1=()->r # captures value of r
f2=(outer r)->r # captures binding to r
f3=(outer r::Int)->r # captures binding with a type assertion

Alternatively, when r does not occur too many times in the function body, one could write

f2=()->outer r 
f3=()->outer r::Int
48

Just to add a data point to how other languages have handled this issue, when Apple introduced blocks to C/Obj-C, they also introduced a __block keyword. The idea is that any variable annotated with __block (e.g. __block int foo) would be auto-boxed if referenced from a closure, and that references would remain live for the lifetime of the closure. In the absence of this keyword, closed-over variables were copied by value into the closure.

Now, given Julia’s history and its status as a dynamic language, I think it would be a shock to users if closed over values were copied by default. But, if I’m understanding all the concerns surrounding this issue, it seems like giving users (or at least advanced users) some amount of manual control over when values are Boxed or not, might be useful.

For example, borrowing from some of the earlier examples, what if using a keyword argument on an anonymous function could trigger explicit non-Boxing behavior:

function f()
    if true
    end
    r::Int = 1
    cb = (;s=r) -> s
    cb()
end
49

It seems like there are two separate issues now being considered in this thread. Here is my summary. Please correct me if it’s wrong.

Issue 1: When a variable, say r is captured, should the r in the closure refer to the same variable in the enclosing scope or to its object? The current Julia language specifies that it refers to the same variable. Changing this language rule would probably be too disruptive. (The suggest by Per to use the outer keyword to distinguish the cases is elegant and nicely consistent with the new syntax, but again, seems disruptive to me.) However, a code can get the “r-refers-to-the-object” behavior by enclosing the closure in a let r=r ... end block. In this case, the parser can assume that the captured variable has a known type AND can eliminate the box. So this solves 15276 and gives a further performance advantage. Does the let trick always eliminate 15276 and the box?

Issue 2: When r is captured, even if we assume that the captured r is the same variable (rather than same object) as in the enclosing scope, can the parser tell that r has a fixed type? Modifying the language to require this behavior is slightly less disruptive but perhaps still undesirable for causing too much breakage. However, a code can get the “r-does-not-change-type” behavior by introducing r before its occurrence in the closure and by fixing its type with a statement like r::T = <init-value-of-r>. So this solves 15276 but does not get rid of the box. Does this kind of r::T assignment always eliminate 15276? What if r is actually an argument to the enclosing function (so that it can’t be declared in this manner)? Then one can say: local r::T=r?

More questions: are the issues for closures identical to the issues for generators? It seems to me that generators are a simpler case because the function implicit in generator syntax is temporary. For example, is there a case involving a generator in which the language cannot assume that the captured variable is a new variable referring to the same object? Besides closures and generators, are there other places where 15276 arises? How about do blocks?

1 Like