Tips to cope with scoping rules of top level `for` loops etc

I am not sure what other scripting languages you are referring to (please be more specific, it could help focus the discussion), but AFAIK Matlab didn’t even have namespaces prior to 2008.

Also, I am not a Matlab expert, but I am not aware of a distinction between what you call “script local” and global.

I am not sure about this. Hard/soft scope was one of the most confusing things about Julia and led to plenty of discussions. Of course, you can say that by the time people understood the issue, they were no longer considered newbies :wink:

If you look back at the discussions and questions, the confusion was almost invariably variables that were unexpectedly local, not about variables that were unexpectedly global. Making more variables local has made that problem worse.

(Even for experienced Julia programmers who understand scope, it’s a continual annoyance to add global keywords for interactive code.)

I still think that the least-bad non-breaking solution at this point would be to default to soft scope for interactive contexts. I haven’t seen a single complaint about IJulia doing this for nearly a year now.

8 Likes

Here’s the thing: intentionally or not, you are working with global variables if you hit this problem. If you want a local-scope-y script, you can just throw a let/end around it. And I’d even be curious how a notebook would feel if each cell was itself a let block. It’d totally wreck most folks’ workflows — including my own — but I wonder how it’d feel to explicitly annotate the global effects and if it’d help prevent some of the more confusing aspects of possible nonlinear cell execution. :slight_smile:

:wave: I strongly dislike it, but it’s mostly because it was a unilateral move. As someone who teaches Julia itself (and not other subjects) through Jupyter notebooks, I dislike that the scope behavior students see is not the scope behavior they get at the REPL and elsewhere. I’ve also seen a handful of posts here and on SO asking why something works in Jupyter notebooks but not in the REPL/Juno/script.

1 Like

Sure, but always complaining about it not working in the REPL. None of those posts were complaining that IJulia should reject their code too—soft scoping is the intuitive behavior.

As for it being unilateral, that’s certainly true. If I’d waited for consensus we’d still be here arguing about it, and meanwhile I need to teach non-programming classes using Julia where explaining the concept of scoping in the middle of a math lecture is impractical. If you’re teaching a programming class, in contrast, most of the code is probably in functions anyway, and you can always turn the soft scoping off. In retrospect I think I made the right choice.

(I don’t think having different scoping rules for interactive code vs. scripts is good, just that it is our least-bad non-breaking option.)

10 Likes

This is good advice for software development, but interactive coding and exploration (where performance is not a concern!) has a place, too.

And if you write all of your code in functions, why do you care about the scoping rules for slow global code? It doesn’t affect you at all. Do we just want to punish people who don’t care about performance and want to play around interactively? Julia should be inconvenient for such users, even if it comes with no benefit for serious software development that doesn’t use global scope much?

4 Likes

Sure, and I get that. It’s kinda like how stopping to explain the concept of IJulia.SOFTSCOPE[] = false gets in the way of teaching what I actually want to teach. :wink:

That’s unfair — there are benefits to the 1.0 rules. It makes it obvious to both humans and static analyzers which identifiers are global. Heck, the Rebugger depends upon it.

2 Likes

Namespaces are largely orthogonal to globals for this sort of scoping, so we can talk about essentially any version of matlab.

It turns out my mental model of matlab is the actual model. Scripts have their own scope. Here is a description of how it works for matlab, which exactly captures my point: the mental model of users of scripts having their own scope is how it is implemented: https://www.mathworks.com/matlabcentral/answers/99602-how-can-i-use-global-variables-and-matlab-workspaces

Now, this implementation itself has its own drawbacks and is not something I am directly suggesting, but it is extremely intuitive for users. If they ever want to share a variable between scripts or non-closure functions, they know they have to define it as a global. It is a very intentional decision, just like it would be in C or compiled languages. i.e. if you ever see a global in matlab, it should smell.

Yes, that is right! By the time people ran into behavior that was confusing in the old v0.6 regime, they were already hooked on julia and could understand the subtlety of the answer. I never saw a discussion on hard/soft scope in Julia from someone who didn’t know what they were talking about.

The reason we are pushing on this again is to resolve that inconsistency. The question is which behavior fit each student’s prior of how scoping should work.

Let me turn it on its head, show me one example of a language supporting scripting style (i.e. top level file with commands not encased in a main() or function to introduce scope, or code in a jupyter notebook) which forces you to mentally think of variables in loops at the top level as being global.

Any example where global is used to annotate a variable inside of a loop inside of a top level file will do. You are going to have to dig deep on this one…

No, you have it wrong. The mental model people have is that the jupyter notebook as a whole has a scope, not each cell. notebook = script = a scope. This is how it is in python, R, matlab, etc. as well. This is also why @stevengj had to hack on top of the whole of IJulia, and why it can’t just introduce a let at a cell level.

That is an implementation detail in Julia, not something essential to how scripting works. As I say above, in matlab the distinction between global and script-level scope is explicit.

I just don’t think you have made the mental shift for why the globals are different there. It is worth trying to understand the perspective before you fight against it so forcefully. Julia is forcing us to use a global instead of a script level scope. That is fine, and it may be the best way to implement it, but intuitively they are used in very different ways.

How about complaints from anyone who hasn’t submitted a PR to julia’s repo, as this feature isn’t for them and I am not sure they have a good sense of how people use a scripting language. If the complaint is “I don’t think Julia should support interactive scripting” then that is fine, but it is a different discussion.

I have no skin in the game from the anger created from unilateral moves (a move I, and many others, strongly supported at the time, even if we weren’t able to pull the trigger ourselves) so maybe we can focus on how best to support current and future users.

Please don’t put words in my mouth. I’m not angry at unilateral moves. I’m not forcefully fighting against you. And I’m certainly not against Julia as an interactive scripting language. I know how scripting workflows work, I’ve used Matlab for over a decade, and I’m aware of the prevailing mental models of Jupyter notebooks. I don’t even think I’ve been very strong in my posts here — I simply wanted to point out the other side of the coin.

3 Likes

OK, my apologies for misunderstanding.

For sure. I think a lot of people here, myself included from my old C++ days, appreciate the value in not having an exception for interactive use at the top level. You have all done a good job of pointing out the other side of the coin, and for non-interactive/exploratory use-cases, your position is certainly correct.

It sounds to me, though, that if we took @stevengj 's suggestion to make the softscope an option at the script level (and possibly the REPL, though I care less about that) the side effects would be (1) experts would have more difficulty reasoning about scope in their code at the top level (as opposed to inside of functions, which are not effected); (2) static analyzers operating at the top level of scripts (as opposed to inside of functions, which are not effected) would have more difficulty, or might not even be supportable; and (3) Rebugger in its current form is a casualty.

Have I missed key downsides here? In return, we make the experience of new users (and those journeyman like myself who switch between script and function style) much better, and avoid the current state where users run into (non-intuitive to them) scope issues in the first 5 minutes of using the software.

Is there any way to trigger a discussion to move things forward, as it currently seems like all progress has stalled otherwise. Is having an option, where the default of on/off could be discussed, for softscope on the REPL and script level technically difficult to implement? It seems like the least work. Furthermore, the exact softscope implementation used in IJulia has been used for a year without a single issue that I know of.

That was precisely the original intention of this thread, but taking the scoping rules as they are right now, rather than giving ground for yet another debate about whether they should be changed or not. :angel:

1 Like

Rebugger works fine with IJulia, as I understand it, because the softscope transformation doesn’t touch macro contents.

It looks like there is a chance that let's scoping rule will not be changed (hence not breaking Rebugger): New scope solution

Speaking of let, how about “wrap for in let together with initialization” as another tip?

While I recognize that occasionally people attribute malicious intentions to those who just disagree with them, from you it is somewhat surprising. I hope that the tone if this discussion recovers from this.

No, I don’t want to “punish” anyone. I just think that even from a pedagogical point of view, sugar-coating the issue for newbies harms them in the long run. Scripting is fine if one mostly does assignments, but nontrivial control flow (which is the most common case of running into the issue, eg loops) belongs in functions. Otherwise, this inevitably leads to “why is my code so slow” questions.

I recognize that you would like to use Julia as a scripting language in teaching where what you call “software development” (which looks like any kind of structured approach beyond trivial scripts) is outside the scope of the course. I understand that it would be convenient for you to do this as you are an expert in Julia. I just don’t think that Julia is well-suited for this, and that we should bend it in this direction.

1 Like

Sorry, I didn’t mean to make you sound malicious.

When you say you don’t want to “sugar-coat the issue” because “nontrivial control flow … belongs in functions,” it sounds like you want working in global scope to be hard as an end in itself. I fundamentally disagree with this viewpoint — we shouldn’t make sacrifices in Julia usability unless it has a clear benefit for production code, which is not the case here (“serious” code in functions is not affected by the scoping rules for global loops).

(I don’t quite understand why you are so passionate about a language rule that, from the sound of it, is irrelevant to the code that you write.)

If we want to make it harder to write slow code in Julia, there are lots of things we could do. For example, we could require a special annotation for type-unstable functions. I don’t think we want to go there.

2 Likes

I think there is a misunderstanding. Doing experiments with created or imported data at REPL is not ‘using Julia as a scripting language’ - actually it’s an interactive way to work. btw: you’ll find “We want it interactive and we want it compiled.” in the " Why We Created Julia".

I work in REPL mostly in matlab, sometimes in python and moving back to an edit-compile-run workstyle would be (yes) moving back in time and experience. Along that i also disagree that fixed length loops (for loops) are a non-trival control flow. In matlab all that implicit indexing stuff (find / logical indices) enable to work without it (when your data has reactangular shape) - while in julia the method of choice seems to be explicit loops.

If julia is not suited to do serious work at REPL, then -well- i’ll move on.

3 Likes

Just to clarify: I think of the combination of nontrivial control flow (for, while, etc) and global state as something that should be possible and has some uses, but I don’t think the language should encourage it as it runs into other issues (mainly performance traps, which are at least equally confusing for newbies).

“Sugar-coating” here refers to the strategy of making a certain kind of coding style easier for newbies, which then, if adopted, runs into inevitable problems later on. I understand that the intention is to help newbies, but I just don’t think it does. This is why I care about this, even though — you are correct — personally I am not much affected as I avoid this kind of code implicitly.

Also, I don’t think that the cost of the current rules is too high, either. It is my impression that people just bump into this once and learn about it, like many other things. But again, it is possible for reasonable people to disagree about this, as we don’t have hard data.

That said, this issue is not black and white for me. I recognize your POV as valid, too, I think we just have a different weight on certain things. Also, I trust the core devs to make the right choice about this eventually. If they adopt a “friendly” version of global scope for scripting that is consistent and easy to reason about, I am fine with that.

We do, however, have the experience of every other dynamic language that is widely used interactively, and none of them behave the way Julia does with respect to scoping of global loops.

(Julia’s design has always been to make as few sacrifices to usability/productivity as possible, compared to other dynamic languages, in order to allow high performance. Not to make sacrifices to try to enforce high performance.)

It’s not just “newbies”, it’s also experienced programmers who want to explore interactively some of the time. Not all code is performance-critical, nor should all code be optimized for performance!

What it comes down to is that the main “benefit” people are citing for the global scoping rules is that it makes commonplace interactive (but slow) code harder to write, not that it makes life significantly easier for “serious” coding. I don’t see this as a compelling argument.

We can’t change the rules now, but we can still improve the interactive situation.

11 Likes

@Tamas_Papp I object to your always repeated assertion that only newbies would ever write loops at global scope. To give some examples:

  1. @benchmark always demands top-level scope (no function!). Hence, lots of ugly annoying global annotations if you want to fill a table with a few 100 benchmark results. Generating + plotting + archiving some data comparing different algorithmic choices and scaling behavior is imo good practice and not “just for newbies”.

  2. Batch processing a bunch of problem instances over night. Of course the functionality is inside proper functions and compiled. But in the end, there is some code-vomit-style script that calls all these optimized functions. Sure you could work to write beautiful scripts, but time is better spent writing beautiful algorithms.

  3. Interactive exploration of data / problems. That’s what jupyter notebooks are super cool for! You want no scoping problems here, and don’t care about perf (because there always is a function boundary before hitting expensive stuff).

Most of my code and workflow is unaffected by global scope questions. But there are literally zero cases where I as a user think “oh, how nice that we have this new scope rule, so much better than 0.6”; instead, it is a constant low-level gripe.

@jlperla gave a perfect explanation of the mental model many people: Top-level code has an enclosing scope, just as every other piece of code, and the same rules apply (for top-level code, the enclosing scope happens to coincide with module-level global scope). This is a perfectly sensible mental model and language design. Julia does something else (also perfectly sensible), where the global-local dichotomy is more pronounced than a “hierarchy of scopes”. I think that everyone here should try to understand both models, not just the one currently chosen in julia.

6 Likes

I think the thread is well-intentioned, but there is a much deeper issue here. The basic consensus among people who do lots of scripting seems to be: (1) Tell beginners, who invariably write script-style code, to start in IJulia; (2) avoid the REPL or julia files for scripting-style entirely and only work with non-script code (i.e. everything in functions). If you tell them that code at the top level simply won’t match their intuition, it will save them confusion until they understand Julia well enough that this isn’t a problem.

I think that making the “tips” more complicated will likely confuse them further. I think having a single, simple tip is the easiest. Explaining a let to someone who doesn’t understand the subtlety of scoping, isn’t going to help.

Please don’t put words in my mouth, I never said this. The pedagogical motivation is prominent for soft global scope though, which is why it was repeatedly used as an argument in favor of it (and I think it is one of the best arguments), and why I used it as an example. This does not mean other uses are not relevant.

I am now going to withdraw from this discussion for a while since I am not comfortable with its tone any more. I understand that there are people who feel passionately about the issue, but I consider it a technical one best discussed in a dispassionate manner. I am afraid that this is becoming very difficult now.

1 Like