New scope solution

[tim.holy]

are there examples of for or let blocks evaluated in a module scope where we definitely would want the variables in those blocks to be local?

Yes, this happens frequently when you copy/paste method bodies to the REPL and wrap them in a let. For example, in a function I can use first as a variable without conflicting with the function named first; if first is local to the let, then there is also no conflict and my copy/paste debugging works. If in contrast first is global, I have to change the name of that variable before I can debug in this fashion.

(@StefanKarpinski, I’ve not yet had time to really sit down and think about this carefully; I will try to do so soon, though I suspect you’ll get all the feedback you need from others. No matter what, kudos for coming up with devious solutions!)

[StefanKarpinski]

@jballanc

Forgive me if I’m missing something, but couldn’t we perhaps have an even simpler heuristic: variables in non-function scopes are global when they exist outside of a function scope?

This was precisely the solution previously proposed here:

Another possible solution to the global scope debacle

The infamous issue https://github.com/JuliaLang/julia/issues/28789 has been on most of our minds for a while now. Although I’ve stopped posting there, I can tell you I’ve been thinking about it a lot, as have Stefan and Jameson and others, and we have continued discussing it off and on. We should really try to get it right. Last time around, we more-or-less settled on changing only the REPL, which has the main advantage of being non-breaking for most code. But after sleeping on it, it does not …

[StefanKarpinski]

@Tamas_Papp

• for , while , try - catch - finally , let , comprehensions, broadcast-fusing,

All scope-introducing constructs that occur outside of a function body.

• only for the REPL? or all module / baremodule ?

Both. One of the goals is not to have different behaviors in different places. Since that inevitably leads to a series of questions about “why does this work in the REPL but not in my script/module.” In particular, since “scripts” (aka programs) operate in Main, which set of rules should apply to them? Are they interactive or non-interactive?

[swissr]

@StefanKarpinski

One of the goals is not to have different behaviors in different places.

[jballanc]

Ok, yeah so I did miss something…apologies! In my defense, it’s been a bit difficult as a bystander to try and follow all the proposed solutions and counter-examples. Glad we have some individuals focusing so much of their attention on this issue. From the multiple issues/threads, it’s obviously something that’s hit a nerve and should be carefully considered…so thanks!

[jballanc]

@tim.holy

For example, in a function I can use first as a variable without conflicting with the function named first ; if first is local to the let , then there is also no conflict and my copy/paste debugging works.

So, this has me wondering how much of the trouble around this issue and proposed solutions has to do with scoping, and how much has to do with shadowing? I wonder if it’s worth considering these as separate issues? Maybe we turn Julia into a LISP-2 in v2.0? (…ok, that last one’s a joke )

[StefanKarpinski]

@foobar_lv2

Forgive me if I’m wrong, but I thought for and while are syntactic sugar for let and conditional @goto (plus some iterate calls in the case of for ); and if , else , etc are syntactic sugar for conditional @goto .

For that reason, the rule needs to work post-lowering and be clear on this lower level […]

This transformation must operate very early on AST, before lowering. For technical reasons, it has to since it has to come before closure conversion. But more significantly, it would also be very broken if the behavior of such a feature depended on details of lowering; lowering is an implementation detail, not a feature of the language.

[StefanKarpinski]

How are these different? If a local name shadows a global one, doesn’t that mean that it has local scope? Note that talking about this issue being about scope is a bit misleading: no scopes are changed on any of the proposals. The only question is about automatic inference of local and global annotations when x = is seen in various contexts.

(Also, Jeff and I both found the Lisp-2 joke very amusing )

[Tamas_Papp]

@jballanc

Maybe we turn Julia into a LISP-2 in v2.0?

Only allowing the introduction of new variables with either a let (more or less equivalent to cl:let) or a global (= cl:defparameter) would also solve this problem, both technically and then socially (everyone would then beg the core devs to return to the 1.0 semantics, and no sane person would bring up the issue again for fear of causing something similar).

[mohamed82008]

@StefanKarpinski

Examples where a global is used in a write-only fashion inside of a loop require an explicit global annotation in order not to error

Does this include for loop variables which are not used inside the loop (or a comprehension)?

println("Wake up")
for i in 1:3
    println("5 more seconds")
    sleep(5)
    println("Wake up")
end 
println("Fine!")

[StefanKarpinski]

@mohamed82008

Does this include for loop variables

No, loop variables are always local.

[jballanc]

@StefanKarpinski

How are these different?

If the question is: “Why not allow variables in for loops to always have global scope?”, it seems like the answers fall into one of two categories:

1. Because we would have to be more careful with the names we use, since a re-definition of a global variable within a for loop would last for the duration of the script/REPL-session (shadowing).
2. Because this would generate a lot of values and objects that might consume considerable memory and never go out of scope to be collected (scoping).

Obviously, an elegant solution that manages to address both categories would be desirable, but if it proves difficult to “kill two birds bugs with one stone patch”, perhaps addressing these concerns separately could yield better results? An example I provided…somewhere in one of these threads…was how Ruby managed to address the issue of shadowing in lambdas, while still having many unresolved problems around scoping.

[Ultima]

@jballanc

Obviously, an elegant solution that manages to address both categories would be desirable, but if it proves difficult to “kill two birds bugs with one stone patch”, perhaps addressing these concerns separately could yield better results?

In my humble opinion of a discussion that excludes me due my lack of experience in the intricacies, i could imagine a rather layman’s solution:
JULIA already has a: default REPL mode, a Shell mode, a package mode, … ? Help-mode … , why not having the best of both worlds in Julia with a default REPL mode → julia> plus a quick prototype mode that enables copy and paste from forums, copy-paste module code etc… Even a copy-paste from a julia> mode would not collide with the proto> mode in case ist contains declarations of globals within for loops etc. cause in proto> mode the declaration of a global within code would be redundant, but the proto> mode code wouldn’t run in the default Julia> Repl without throwing an error.
But…, i have no clue wether this “best of both worlds” a Julia 1.0 mode next to a old-school proto> 0.6 mode within the REPL is doable, or even disruptive.

[StefanKarpinski]

That’s certainly possible. The main issue is that people will then be confused about why their code works one way in one place and differently in another.

[mbauman]

It’s worth emphasizing that the Julia 0.6 behavior also didn’t seamlessly allow copy-paste of code from local scope to global scopes, either. See Tim’s first example above.

[s-broda]

FWIW, I fear that the cure proposed here may be worse than the disease. It seems to me that fundamentally, the confusion out of which all the discourse/stackoverflow questions arise stems from the fact that the meaning of “x=1” depends on the context in which it appears (at top level scope or in a loop). The solution proposed here (whether the original one or its modifications) makes this context dependence even more subtle; while it addresses the most common cases in a DWIM way, these subtleties may well lead to even more confusion in more complex/corner cases down the road.

In the spirit of explicit is better than implicit, maybe the following, perhaps radical, solution is a way to address the fundamental issue: KEEP the current (1.0) behavior, except for one change: declaring or modifying, but not reading, a global variable ALWAYS requires a global annotation; hence x=1 at top level scope is an ERROR (with an exception for functions perhaps). x=1 in local scope ALWAYS creates/modifies a local.

The advantage of this is its simplicity, which leaves very little room for confusion. Beginners could just be told that this is how variables are declared in Julia. If they try the likes of

x=1
for i=1:10
    x += 1
end

they still get an error, but the error now appears at “x=1”, with an explicit hint to use global. This is a lot less confusing than the current x not defined. Similarly, code pasted from a function will still not work when pasted to the REPL, but at least you get a very clear error message. A possible knock-on effect is that this might reduce the number of issues where people benchmark in global scope.

[mohamed82008]

Taking away the right to x = 1, with no global, sounds like a step down the usability ladder to me. If you come from another scripting language, this will be really annoying.

[s-broda]

In my view Julia has evolved beyond a scripting language. And globals are special; see the numerous performance questions from people benchmarking in global scope. Syntactically singling them out as such is probably worthwhile.

Note also that in Matlab, IIRC accessing a global from local scope (a function) also requires the global keyword. Matlab just has fewer local scope introducing constructs. This doesn’t seem to have caused many issues.