Why not an even-harder scope?

Functions and const variables appear to be already treated differently, so I am not really sure if that is a reason for that final behavior. I wander if it would be possible to create a macro like @global_warn f().. which would throw an error, or warning, if any non-constant global is used inside the function. That would have its uses, particularly for didactic purposes. I am not sure how different would that be from the current @code_warntype, though.

One lesson from all those scope discussions is that scoping rules have complex interactions with each other and the rest of the language, so gotchas like this are common. Ideally new proposals should at least work through the 30–50 relevant examples that came up in those, and/or the cases in the tests.

4 Likes

Now that it’s clear it’s too much of a hassle to declare outer to use any outer variable (it would even include methods), I’m thinking what if outer was required to reassign an outer variable. Basically every variable in a scope that is assigned a value is treated as if it was declared local by default.
On one hand, still needs the outer line but now just for the variables that get reassigned. On the other, code pasted from inside a function to global scope both use the outer line, no need for different interactive behavior to avoid global edits.

I’ve considered that design before—in which case you really want the boundary where an outer is required to be function (and probably let), since requiring outer to assign to a local from a loop would be very confusing and irritating. The biggest issue with this approach is that it makes closures behave quite differently from other scope constructs and there are a fair number of situations where inner functions really want to behave like any other block. Consider this loop:

function f(v)
    s = zero(eltype(v))
    for x in v
        s += x
    end
    return s
end

There’s a variant that uses foreach and a do-block closure:

function f(v)
    s = zero(eltype(v))
    foreach(v) do x
        s += x
    end
    return s
end

With the current design, these behave the same: because s is already a local variable in f when s += x happens, that existing local is updated. With the proposed outer rule, you’d have to remember to change the foreach version to this:

function f(v)
    s = zero(eltype(v))
    foreach(v) do x
        outer s += x
    end
    return s
end

In this case there’s not so much reason to want to use the foreach version here, but there are other cases where it’s common to want to switch back and forth between closure and non-closure versions. Consider how common it is to start with something like this:

function g(path, name)
    f = open(path)
    name = uppercasefirst(name)
    println(f, "Hello, $name.")
    close(f)
    return name
end

That works well enough, but if the print statement errors for some reason, the file doesn’t get closed until it gets garbage collected, which isn’t the best form, so instead it’s considered a best practice to use the do-block form of open, like this:

function g(path, name)
    open(path) do f
        name = uppercasefirst(name)
        println(f, "Hello, $name.")
    end
    return name
end

This version guarantees that f is closed when the open block exists, regardless of how it exits (normally or by throwing an error). With the proposed outer rule, this would have to be written like this instead:

function g(path, name)
    open(path) do f
        outer name = uppercasefirst(name)
        println(f, "Hello, $name.")
    end
    return name
end

If you forgot to put the outer annotation in there, you would get an undefined reference error for the usage of name in uppercasefirst(name), which is the global scope problem all over again, but in a situation where we cannot really print a warning, both because it’s slow, but also because, in general, shadowing a local with another local of the same name is a perfectly reasonable thing to do. Compare this with shadowing a global with an implicit local outside of a function body (i.e. in soft scope), which is definitely a bad idea, never performance-sensitive, and is where we currently print a warning.

The current design minimizes the difference between closures and normal scope blocks like loop bodies, which seems like a pretty significant consideration. I think it’s pretty important in day-to-day Julia programming that I don’t have to worry whether wrapping some code in a do-block or other function body affects the behavior of the code.

16 Likes

There is an alternate design suggested by this: do-blocks, instead of being simple syntactic sugar for closures, could become something a bit different—you could make the rule that outer is required to assign to an outer local only in a closure that is syntactically declared with one of the function syntaxes. I.e. there would be a difference between this:

foreach(x -> outer x += x, v) # `outer` required for function syntax

and this:

foreach(v) do x
    s += x # no `outer` required because do blocks aren't function syntax
end

Ruby has already gone down this path: the language has blocks, procs and lambdas which are all slightly different variations on closures with subtle and confusing differences; one of the main purposes of these differences is to make do-blocks behave more like regular code. This is critical in Ruby because it’s idiomatic to write v.each{ |x| s += x } rather than using a for loop, so you really want your closures to be syntactically convenient (although I think the causal relationship probably went the other way). Ruby’s approach to distinguishing these is generally considered to be usable but complex and confusing.

If you’re unhappy about the complexity of Julia’s current global scope behavior, then going the route of Ruby here seems far worse in terms of complexity. Keep in mind that in files, Julia’s rule is dead simple: when x = happens in a scope block, it assigns x if x is already a local, otherwise it creates a new local. That’s it, that’s the whole rule. The only complication at all comes from wanting the REPL to be more convenient, which is where the whole notion of soft scope and ambiguity warnings come in. If you’re not in the REPL, you never have to even think about it. Isolating complexity to the place where interactive convenience requires it seems appropriate; letting it bleed into the entire language because we want convenience in the REPL seems like it would be strictly worse.

13 Likes

Having a lot of time thinking about it, I really believe that our current design is very close to optimal given just these assumptions:

  1. Locals may be implicitly declared by assignment
  2. The rules for all local scopes are the same
  3. The language has closures.

There are languages where all variables have to be explicitly declared, i.e. you have to write var x or local x to declare a new local. This, of course, makes all problems relating to what scope an assignment assigns to go away: it’s the innermost enclosing scope that declares the variable and if there’s none, it’s an error. I would note that you can already program Julia this way if you want to: just declare every local with local x and the language will behave just like this. There are only two issues: you won’t get an error if there is no declaration of a variable and you have to declare a global reference before assigning it from a local scope.

The second assumption precludes rules like the outer one proposed above—we’ve already discussed why it’s desirable to go back and forth between a closure and other scoped constructs like loop bodies, so I won’t repeat that here.

Which brings us to the question: why should loops have scope in the first place? After all, loops don’t have scope in Python. The answer is that Julia, unlike Python, has always had closures and encourages using them. The connection is probably not immediately obvious, but pretty much every time you want to know why some scope thing works the way it does, the answer is “closures”. So let’s work this through step by step. If a language has closures, you can do things like this:

julia> fns = []
Any[]

julia> for i = 1:5
           push!(fns, () -> i)
       end

julia> [f() for f in fns]
5-element Vector{Int64}:
 1
 2
 3
 4
 5

There are variations on this like using a comprehension instead:

fns = [() -> i for i = 1:5]

But the core issue is the same: you want i to be local to the for loop or comprehension body so that when it gets captured, each closure gets a separate i instead of all of the functions capturing the same i. Suppose we did what Python does and loops didn’t introduce scope at all. We can simulate that here by capturing a global i:

julia> fns = []
Any[]

julia> for ii = 1:5
           global i = ii
           push!(fns, () -> i)
       end

julia> [f() for f in fns]
5-element Vector{Int64}:
 5
 5
 5
 5
 5

Oops. It’s actually even worse than this example suggests since not only is it the same i that is captured in each loop iteration, but it’s global so if it gets assigned after the loop then things go horribly wrong:

julia> fns = Function[]
Function[]

julia> for ii = 1:5
           global i = ii
           push!(fns, () -> i)
       end

julia> i = "oops!"
"oops!"

julia> [f() for f in fns]
5-element Vector{String}:
 "oops!"
 "oops!"
 "oops!"
 "oops!"
 "oops!"

Oops, indeed.

So, if you have closures, loops really must have scope. Python has traditionally dealt with this by refusing to have closures, much to the consternation off everyone who wants to do functional programming in Python. But at some point Python conceded that closures were quite useful and added lambdas, which leads to this unfortunate brokenness in the presence of Python’s scope rules:

>>> fns = [lambda: i for i in range(0,5)]
>>> [f() for f in fns]
[4, 4, 4, 4, 4]

Not great, Bob. And in Python 2, the i is global so it’s really bad:

>>> fns = [lambda: i for i in range(0,5)]
>>> i = "oops!"
>>> [f() for f in fns]
['oops!', 'oops!', 'oops!', 'oops!', 'oops!']

In Python 3 they fixed this by making i local to the comprehension (although local to the whole thing, not a new local for each iteration, so you still capture the same local i five times). And even in Python 3 you still have this trap:

>>> fns = []
>>> for i in range(0,5):
...     fns.append(lambda: i)
...
>>> i = "oops!"
>>> [f() for f in fns]
['oops!', 'oops!', 'oops!', 'oops!', 'oops!']

Because while comprehensions now introduce a local scope, loops still don’t so that i loop variable is not only shared by all the closures, but it leaks out into the global scope where you can clobber it and screw up all of those closures. While it’s good that comprehensions have their own scope, now you have the issue that going back and forth between using a loop and a comprehension is subtly different and may cause bad bugs.

Anyway, this isn’t to pick on Python, but to point out that they don’t have it figured out—scope is a pretty huge footgun in Python. And, more importantly for Julia, to show why a language that has closures really needs to have pretty granular scopes, in particular including loops and comprehensions.

The only thing that might be appealing to change in Julia 2.0 would be to make behavior in files to match the REPL but still print a warning in the case where an implicit local shadows a global of the same name. That would essentially bring back full 0.6 rules with the addition of a warning in the case where a local implicitly shadows a global—which is precisely the case where 1.0 rules differ from 0.6 rules. That way the behavior would be the same everywhere (albeit with the more complex 0.6 rules), with an extra warning in an ambiguous case in a file while the REPL would be slightly more lenient and allow you to assign a global from a loop without needing to declare the assignment to be global to avoid the warning.

24 Likes

Sounds perfect. I think you guys found the only two local optima in the design space. 0.6 vs 0.7 rules,so eliminating the convex combination makes sense. Maybe there is a new scope level which could expand the dimensions to work with, but it isn’t clear one exists - especially since Julia doesn’t depend on files the same way python and matlab do.

The one thing I would add is that it seems like all of these legitimate bugs you uncovered with the v0.7 rules could be covered by linting or at least a toggle on the Julia executable. I don’t see why having a --check-global-scope=true which everyone would want on for unit testing wouldn’t make everyone happy. Scripters, IJulia, and the interactive environments like vscode would could flip the switch.

2 Likes

Very cool to see how a design choice can have far-reaching consequences, I didn’t even think of do-blocks and closures at all.
I will point out that my "outer needed to reassign outer variables" idea would also require an outer line in the for-loop version of f(v), so at least it’d be consistent with the foreach version. But what really sold me on the local scope variable-sharing is one-line functions. Not a whole lot of room for an outer line there.

That is certainly consistent but at that point you’d be better off just requiring explicit declaration of all variables.

3 Likes

Yeah, at which point it will no longer feel like Julia.
Now I’m feeling it’s less the scoping rules and more that people want to swap code between a function body and the REPL, which unfortunately is a global scope, so tweaking the REPL was the right call. I’m tempted to suggest tweaking the REPL to be able to do a convenient version of pasting code into a throwaway function and running it, but I’m very wary that’ll have some far-reaching consequences too.

1 Like

Doing that would provide the ideal opportunity to do all the other breaking changes, and switch the surface syntax to S-expressions.

It would be painful and horribly breaking, but would get lost in the discussion about needing to declare variables.

8 Likes

Thanks for the explanation Stefan!

I got confused at your example of constructing a vector of closures. How does i being local in the loop cause each closure to capture a different value of it? Consider the following example:

julia> function g()
           fns = []
           let i = 1
               while i ≤ 5
                   push!(fns, () -> i)
                   i += 1
               end
           end
           fns
       end
g (generic function with 1 method)

julia> [f() for f in g()]
5-element Array{Int64,1}:
 6
 6
 6
 6
 6

Surely i is local in the let block, but all the closures capture the same value. What makes a closure capture either the value of a variable or a reference to it?

There is a single i variable here, declared by let, which is captured five times. In a for loop, each iteration of the loop has its own scope and any variables that are local each iteration — including the iteration variables — are new on each iteration. If you wanted the same effect in a while loop, you’d have to do this:

function h()
    fns = []
    ii = 1
    while ii ≤ 5
        i = ii # local to this iteration of the loop
        push!(fns, () -> i)
        ii += 1
    end
    return fns
end
5 Likes

I noticed that @A_Dent kept saying “closure capture the value”, but just to make sure, closures captures variables, not the objects, right (the docs seem to say so)? If they captured objects, it doesn’t seem like you even need scopes and new variables each iteration to avoid the repeated value bug, you could just inline the integers into the function.
Now that I think of it, why don’t closures capture values instead of variables?

2 Likes

It’s a good question and I agree it that on the surface it seems like it’d make things simpler, but it’s actually an exception in how scopes generally work. The crux is that you want all of these constructs to behave the same:

julia> function f1()
           x = 1
           for i in 1:10
               x = x+1
           end
           return x
       end
f1 (generic function with 1 method)

julia> function f2()
           x = 1
           [(x = x+1) for i in 1:10] # surprise, this is a closure!
           return x
       end
f2 (generic function with 1 method)

julia> function f3()
           x = 1
           map(i->(x = x+1), 1:10)
           return x
       end
f3 (generic function with 1 method)

julia> function f4()
           x = 1
           g(i) = (x = x+1)
           map(g, 1:10)
           return x
       end
f4 (generic function with 1 method)

julia> f1() == f2() == f3() == f4() == 11
true

Indeed, if closures — and functions generally — captured by value then you’d no longer be able to use non-constant globals! The way I keep my mental model straight is by remembering that closures aren’t special. If they share a name with an outer scope, then that name is consistently used. In the example above, I’m simultaneously using the value of a name and changing what that name should identify, but you could separate the two:

julia> function f5()
           x = 1
           g = () -> x+1
           for i in 1:10
               x = g()
           end
           return x
       end
f5 (generic function with 1 method)

julia> f5()
11
5 Likes

That’s a design direction to think about. If we accept that we want loop scopes and closure scopes to behave similarly so that we can freely move back and forth between for and foreach and comprehensions (which are secretly closures), then let’s think through the consequences. Suppose we have this for loop:

t = 0
for i = 1:10
    t += i
end

Based on the equivalence assumption, this should behave the same way as this:

t = 0
foreach(1:10) do i
    t += i
end

Now, if closures capture variables by value, then t in the closure is a new t that is assigned to the initial value of the outer t value. That means that the outer t is unaffected by the assignment in the closure / loop body, so the effect of the entire for loop or foreach call is nothing: the outer t is never changed and at the end it’s still zero.

It seems pretty clear that the vast majority of potential users would not be happy with that. So either this approach is a dead end or we decide to break the assumption that loops and closures behave similarly, in which case the for loop version could modify t while the foreach version could leave it unchanged. Personally, I think that a lot of the utility of closures is that they allow you to do things like implement a for loop with user-defined code, so I’m very reluctant to break that kind of principle.

5 Likes

So in short, closures (like any top-level block) capture variables because that’s the only way they can make any changes to those variables. The only case where it seems easier to capture values is when the value of that variable never changes, like for i = 1:5 push!(fns, () -> i) end.

Right. That’s an interesting observation: capture by value does generally seem more convenient/intuitive when you only want to read a values from an outer scope inside a closure; when you want to modify a value outside a closure, then capture by value doesn’t work, since you can’t modify the outer binding. The motivations for small scopes generally only come from the former use case (reading but not writing outer variables).

3 Likes

Something I don’t like about the implicit capturing of variables in outer scopes is that it allows for this kind of bugs to slip in:

function check_computation()
    function computation(a, b)
        # ... some difficult computations ...
        result = a + b + 1 # bug: we were meant to return `a + b`
        # ... some other computations
        return result
    end

    a, b = 10, 20
    result = a + b # the expected result, computed in some independent way to double-check
    computed = computation(a, b)

    if result == computed
        print("They agree:\nexpected = $result\ncomputed = $computed")
    else
        print("They disagree:\nexpected = $result\ncomputed = $computed")
    end
end

I think that at first it may be surprising to realize that check_computation() prints the erroneous

They agree:
expected = 31
computed = 31

and does not detect the bug in computation. The behavior would be different if computation were defined outside the scope of check_computation: in that case the bug would be detected with the output

They disagree:
expected = 30
computed = 31

This makes it very hard, almost impractical, to refactor and move functions in and out of other functions scopes without introducing subtle bugs. The only safe way I can see is to declare local every variable of an inner function (if we don’t want to intentionally capture it, of course).

2 Likes

The only safe way I can see is to declare local

Yes, but what’s the point? Evidently you’re nesting a function because you want/need enclosing scope for some reason, which means you also accept the risk of accidentally capturing unintended variables.

There seem a couple ways around this. First, you could and should just say local just as you say. Second, you could elect not to nest the function, and instead have a separate computation that receives all its information as explicit arguments. That would be safe, just a bit annoying, although you could just define a closure in check_computation.

Nobody forces you to use nested functions, and when you decide to, unfortunately you have to accept the risks. I generally avoid nesting except for very short functions, preferably anonymous, where there aren’t a lot of extra (implicitly local) variables floating around, and it’s clear that the intent is some sort of closure.

3 Likes