Why not an even-harder scope?

Internals & Design
1

I know this is hardly the first post about scoping being “different” in different places (documented here), but I had a thought and couldn’t really find any post that addressed it.

I’ll try to give a short recap: version 1 tried to simplify things by going full hard scope, and it’s pretty compelling: nobody likes their code to change with an addition of a faraway global variable. However, it made copying and pasting code from inside a method to the REPL a hassle because you had to add the global keyword to use the variables outside the block (and remove it to paste it back in a function), so soft scope made a comeback to the REPL (and other interactive environments).

In fact, reading the docs started making me uncomfortable with blocks implicitly using any outer variables, even local variables in a method. Then I noticed that there is one exception. The iteration variable in a for-looper’s header never uses outer local variables by default, needing the outer keyword to do so. That led to my thought: instead of trying to strike a balance between hard and soft scope, why couldn’t we just go even-harder scope and not allow blocks to use outer variables without outer? This seems like a much easier rule to avoid both the faraway global issue and the copy-pasting issue, and the one hassle is a very explicit line in the block like outer x, y, z naming all the outer local/global variables to check for. The explicit line also helps make it clearer what variables you have to keep track of outside the block. I feel like I must be missing something because I feel someone would’ve mentioned it by now.

1 Like
2
1 Like
3

I am aware that version 1 commits to stability, so I’m not expecting a such a fundamental (and featureless) syntax change to happen in future versions. I meant this question more in past tense: the even-harder scope seems to me like an obvious idea someone would have thought of already (the outer keyword is already there), so I’m guessing there is an equally obvious drawback that I just can’t think of.

1 Like
4

Seems super annoying, no?

8 Likes
5

I think one of the reasons is that there is no much difference between a variable that addresses a function and any other variable that addresses a value, or vector. Therefore, with that hard scope one would need to declare even all the functions which are being used inside each other function. Of course, this would be impossible.

In Fortran, for a while variables declared with specific names had by default specific types (i as Integer, x as Float), etc. To avoid errors, the implicit none flag was invented, which can be added to a function to guarantee that all variables where declared and no implicit assumptions were made. It might be possible to create some flag of that sort, or a macro, to warn the coder of the of global variables, perhaps.

1 Like
6

Personally, writing that extra first line would be a minor hassle to me because it feels similar to listing variables in a method/struct definition, and it would save me some time checking which variables came from outside the block. I do this much with a comment, though.
EDIT: actually seems like a huge hassle now that it’s been pointed out that method names are (const) variables

7

You’re right, needing to write outer println over and over would legitimately be annoying. This could be mitigated by not requiring outer for calls println("hello") or const variables, but that doesn’t cover outer variables like x = println.

8

Functions and const variables appear to be already treated differently, so I am not really sure if that is a reason for that final behavior. I wander if it would be possible to create a macro like @global_warn f().. which would throw an error, or warning, if any non-constant global is used inside the function. That would have its uses, particularly for didactic purposes. I am not sure how different would that be from the current @code_warntype, though.

9

One lesson from all those scope discussions is that scoping rules have complex interactions with each other and the rest of the language, so gotchas like this are common. Ideally new proposals should at least work through the 30–50 relevant examples that came up in those, and/or the cases in the tests.

4 Likes
11

Now that it’s clear it’s too much of a hassle to declare outer to use any outer variable (it would even include methods), I’m thinking what if outer was required to reassign an outer variable. Basically every variable in a scope that is assigned a value is treated as if it was declared local by default.
On one hand, still needs the outer line but now just for the variables that get reassigned. On the other, code pasted from inside a function to global scope both use the outer line, no need for different interactive behavior to avoid global edits.

12

I’ve considered that design before—in which case you really want the boundary where an outer is required to be function (and probably let), since requiring outer to assign to a local from a loop would be very confusing and irritating. The biggest issue with this approach is that it makes closures behave quite differently from other scope constructs and there are a fair number of situations where inner functions really want to behave like any other block. Consider this loop:

function f(v)
    s = zero(eltype(v))
    for x in v
        s += x
    end
    return s
end

There’s a variant that uses foreach and a do-block closure:

function f(v)
    s = zero(eltype(v))
    foreach(v) do x
        s += x
    end
    return s
end

With the current design, these behave the same: because s is already a local variable in f when s += x happens, that existing local is updated. With the proposed outer rule, you’d have to remember to change the foreach version to this:

function f(v)
    s = zero(eltype(v))
    foreach(v) do x
        outer s += x
    end
    return s
end

In this case there’s not so much reason to want to use the foreach version here, but there are other cases where it’s common to want to switch back and forth between closure and non-closure versions. Consider how common it is to start with something like this:

function g(path, name)
    f = open(path)
    name = uppercasefirst(name)
    println(f, "Hello, $name.")
    close(f)
    return name
end

That works well enough, but if the print statement errors for some reason, the file doesn’t get closed until it gets garbage collected, which isn’t the best form, so instead it’s considered a best practice to use the do-block form of open, like this:

function g(path, name)
    open(path) do f
        name = uppercasefirst(name)
        println(f, "Hello, $name.")
    end
    return name
end

This version guarantees that f is closed when the open block exists, regardless of how it exits (normally or by throwing an error). With the proposed outer rule, this would have to be written like this instead:

function g(path, name)
    open(path) do f
        outer name = uppercasefirst(name)
        println(f, "Hello, $name.")
    end
    return name
end

If you forgot to put the outer annotation in there, you would get an undefined reference error for the usage of name in uppercasefirst(name), which is the global scope problem all over again, but in a situation where we cannot really print a warning, both because it’s slow, but also because, in general, shadowing a local with another local of the same name is a perfectly reasonable thing to do. Compare this with shadowing a global with an implicit local outside of a function body (i.e. in soft scope), which is definitely a bad idea, never performance-sensitive, and is where we currently print a warning.

The current design minimizes the difference between closures and normal scope blocks like loop bodies, which seems like a pretty significant consideration. I think it’s pretty important in day-to-day Julia programming that I don’t have to worry whether wrapping some code in a do-block or other function body affects the behavior of the code.

16 Likes
13

There is an alternate design suggested by this: do-blocks, instead of being simple syntactic sugar for closures, could become something a bit different—you could make the rule that outer is required to assign to an outer local only in a closure that is syntactically declared with one of the function syntaxes. I.e. there would be a difference between this:

foreach(x -> outer x += x, v) # `outer` required for function syntax

and this:

foreach(v) do x
    s += x # no `outer` required because do blocks aren't function syntax
end

Ruby has already gone down this path: the language has blocks, procs and lambdas which are all slightly different variations on closures with subtle and confusing differences; one of the main purposes of these differences is to make do-blocks behave more like regular code. This is critical in Ruby because it’s idiomatic to write v.each{ |x| s += x } rather than using a for loop, so you really want your closures to be syntactically convenient (although I think the causal relationship probably went the other way). Ruby’s approach to distinguishing these is generally considered to be usable but complex and confusing.

If you’re unhappy about the complexity of Julia’s current global scope behavior, then going the route of Ruby here seems far worse in terms of complexity. Keep in mind that in files, Julia’s rule is dead simple: when x = happens in a scope block, it assigns x if x is already a local, otherwise it creates a new local. That’s it, that’s the whole rule. The only complication at all comes from wanting the REPL to be more convenient, which is where the whole notion of soft scope and ambiguity warnings come in. If you’re not in the REPL, you never have to even think about it. Isolating complexity to the place where interactive convenience requires it seems appropriate; letting it bleed into the entire language because we want convenience in the REPL seems like it would be strictly worse.

13 Likes
14

Having a lot of time thinking about it, I really believe that our current design is very close to optimal given just these assumptions:

  1. Locals may be implicitly declared by assignment
  2. The rules for all local scopes are the same
  3. The language has closures.

There are languages where all variables have to be explicitly declared, i.e. you have to write var x or local x to declare a new local. This, of course, makes all problems relating to what scope an assignment assigns to go away: it’s the innermost enclosing scope that declares the variable and if there’s none, it’s an error. I would note that you can already program Julia this way if you want to: just declare every local with local x and the language will behave just like this. There are only two issues: you won’t get an error if there is no declaration of a variable and you have to declare a global reference before assigning it from a local scope.

The second assumption precludes rules like the outer one proposed above—we’ve already discussed why it’s desirable to go back and forth between a closure and other scoped constructs like loop bodies, so I won’t repeat that here.

Which brings us to the question: why should loops have scope in the first place? After all, loops don’t have scope in Python. The answer is that Julia, unlike Python, has always had closures and encourages using them. The connection is probably not immediately obvious, but pretty much every time you want to know why some scope thing works the way it does, the answer is “closures”. So let’s work this through step by step. If a language has closures, you can do things like this:

julia> fns = []
Any[]

julia> for i = 1:5
           push!(fns, () -> i)
       end

julia> [f() for f in fns]
5-element Vector{Int64}:
 1
 2
 3
 4
 5

There are variations on this like using a comprehension instead:

fns = [() -> i for i = 1:5]

But the core issue is the same: you want i to be local to the for loop or comprehension body so that when it gets captured, each closure gets a separate i instead of all of the functions capturing the same i. Suppose we did what Python does and loops didn’t introduce scope at all. We can simulate that here by capturing a global i:

julia> fns = []
Any[]

julia> for ii = 1:5
           global i = ii
           push!(fns, () -> i)
       end

julia> [f() for f in fns]
5-element Vector{Int64}:
 5
 5
 5
 5
 5

Oops. It’s actually even worse than this example suggests since not only is it the same i that is captured in each loop iteration, but it’s global so if it gets assigned after the loop then things go horribly wrong:

julia> fns = Function[]
Function[]

julia> for ii = 1:5
           global i = ii
           push!(fns, () -> i)
       end

julia> i = "oops!"
"oops!"

julia> [f() for f in fns]
5-element Vector{String}:
 "oops!"
 "oops!"
 "oops!"
 "oops!"
 "oops!"

Oops, indeed.

So, if you have closures, loops really must have scope. Python has traditionally dealt with this by refusing to have closures, much to the consternation off everyone who wants to do functional programming in Python. But at some point Python conceded that closures were quite useful and added lambdas, which leads to this unfortunate brokenness in the presence of Python’s scope rules:

>>> fns = [lambda: i for i in range(0,5)]
>>> [f() for f in fns]
[4, 4, 4, 4, 4]

Not great, Bob. And in Python 2, the i is global so it’s really bad:

>>> fns = [lambda: i for i in range(0,5)]
>>> i = "oops!"
>>> [f() for f in fns]
['oops!', 'oops!', 'oops!', 'oops!', 'oops!']

In Python 3 they fixed this by making i local to the comprehension (although local to the whole thing, not a new local for each iteration, so you still capture the same local i five times). And even in Python 3 you still have this trap:

>>> fns = []
>>> for i in range(0,5):
...     fns.append(lambda: i)
...
>>> i = "oops!"
>>> [f() for f in fns]
['oops!', 'oops!', 'oops!', 'oops!', 'oops!']

Because while comprehensions now introduce a local scope, loops still don’t so that i loop variable is not only shared by all the closures, but it leaks out into the global scope where you can clobber it and screw up all of those closures. While it’s good that comprehensions have their own scope, now you have the issue that going back and forth between using a loop and a comprehension is subtly different and may cause bad bugs.

Anyway, this isn’t to pick on Python, but to point out that they don’t have it figured out—scope is a pretty huge footgun in Python. And, more importantly for Julia, to show why a language that has closures really needs to have pretty granular scopes, in particular including loops and comprehensions.

The only thing that might be appealing to change in Julia 2.0 would be to make behavior in files to match the REPL but still print a warning in the case where an implicit local shadows a global of the same name. That would essentially bring back full 0.6 rules with the addition of a warning in the case where a local implicitly shadows a global—which is precisely the case where 1.0 rules differ from 0.6 rules. That way the behavior would be the same everywhere (albeit with the more complex 0.6 rules), with an extra warning in an ambiguous case in a file while the REPL would be slightly more lenient and allow you to assign a global from a loop without needing to declare the assignment to be global to avoid the warning.

25 Likes
15

Sounds perfect. I think you guys found the only two local optima in the design space. 0.6 vs 0.7 rules,so eliminating the convex combination makes sense. Maybe there is a new scope level which could expand the dimensions to work with, but it isn’t clear one exists - especially since Julia doesn’t depend on files the same way python and matlab do.

The one thing I would add is that it seems like all of these legitimate bugs you uncovered with the v0.7 rules could be covered by linting or at least a toggle on the Julia executable. I don’t see why having a --check-global-scope=true which everyone would want on for unit testing wouldn’t make everyone happy. Scripters, IJulia, and the interactive environments like vscode would could flip the switch.

2 Likes
16

Very cool to see how a design choice can have far-reaching consequences, I didn’t even think of do-blocks and closures at all.
I will point out that my “outer needed to reassign outer variables” idea would also require an outer line in the for-loop version of f(v), so at least it’d be consistent with the foreach version. But what really sold me on the local scope variable-sharing is one-line functions. Not a whole lot of room for an outer line there.

17

That is certainly consistent but at that point you’d be better off just requiring explicit declaration of all variables.

3 Likes
18

Yeah, at which point it will no longer feel like Julia.
Now I’m feeling it’s less the scoping rules and more that people want to swap code between a function body and the REPL, which unfortunately is a global scope, so tweaking the REPL was the right call. I’m tempted to suggest tweaking the REPL to be able to do a convenient version of pasting code into a throwaway function and running it, but I’m very wary that’ll have some far-reaching consequences too.

1 Like
19

Doing that would provide the ideal opportunity to do all the other breaking changes, and switch the surface syntax to S-expressions.

It would be painful and horribly breaking, but would get lost in the discussion about needing to declare variables.

8 Likes
20

Thanks for the explanation Stefan!

I got confused at your example of constructing a vector of closures. How does i being local in the loop cause each closure to capture a different value of it? Consider the following example:

julia> function g()
           fns = []
           let i = 1
               while i ≤ 5
                   push!(fns, () -> i)
                   i += 1
               end
           end
           fns
       end
g (generic function with 1 method)

julia> [f() for f in g()]
5-element Array{Int64,1}:
 6
 6
 6
 6
 6

Surely i is local in the let block, but all the closures capture the same value. What makes a closure capture either the value of a variable or a reference to it?

21

There is a single i variable here, declared by let, which is captured five times. In a for loop, each iteration of the loop has its own scope and any variables that are local each iteration — including the iteration variables — are new on each iteration. If you wanted the same effect in a while loop, you’d have to do this:

function h()
    fns = []
    ii = 1
    while ii ≤ 5
        i = ii # local to this iteration of the loop
        push!(fns, () -> i)
        ii += 1
    end
    return fns
end
5 Likes