Tips to cope with scoping rules of top level `for` loops etc

Namespaces are largely orthogonal to globals for this sort of scoping, so we can talk about essentially any version of matlab.

It turns out my mental model of matlab is the actual model. Scripts have their own scope. Here is a description of how it works for matlab, which exactly captures my point: the mental model of users of scripts having their own scope is how it is implemented: How can I use global variables and MATLAB workspaces? - MATLAB Answers - MATLAB Central

Now, this implementation itself has its own drawbacks and is not something I am directly suggesting, but it is extremely intuitive for users. If they ever want to share a variable between scripts or non-closure functions, they know they have to define it as a global. It is a very intentional decision, just like it would be in C or compiled languages. i.e. if you ever see a global in matlab, it should smell.

Yes, that is right! By the time people ran into behavior that was confusing in the old v0.6 regime, they were already hooked on julia and could understand the subtlety of the answer. I never saw a discussion on hard/soft scope in Julia from someone who didn’t know what they were talking about.

The reason we are pushing on this again is to resolve that inconsistency. The question is which behavior fit each student’s prior of how scoping should work.

Let me turn it on its head, show me one example of a language supporting scripting style (i.e. top level file with commands not encased in a main() or function to introduce scope, or code in a jupyter notebook) which forces you to mentally think of variables in loops at the top level as being global.

Any example where global is used to annotate a variable inside of a loop inside of a top level file will do. You are going to have to dig deep on this one…

No, you have it wrong. The mental model people have is that the jupyter notebook as a whole has a scope, not each cell. notebook = script = a scope. This is how it is in python, R, matlab, etc. as well. This is also why @stevengj had to hack on top of the whole of IJulia, and why it can’t just introduce a let at a cell level.

That is an implementation detail in Julia, not something essential to how scripting works. As I say above, in matlab the distinction between global and script-level scope is explicit.

I just don’t think you have made the mental shift for why the globals are different there. It is worth trying to understand the perspective before you fight against it so forcefully. Julia is forcing us to use a global instead of a script level scope. That is fine, and it may be the best way to implement it, but intuitively they are used in very different ways.

How about complaints from anyone who hasn’t submitted a PR to julia’s repo, as this feature isn’t for them and I am not sure they have a good sense of how people use a scripting language. If the complaint is “I don’t think Julia should support interactive scripting” then that is fine, but it is a different discussion.

I have no skin in the game from the anger created from unilateral moves (a move I, and many others, strongly supported at the time, even if we weren’t able to pull the trigger ourselves) so maybe we can focus on how best to support current and future users.

Please don’t put words in my mouth. I’m not angry at unilateral moves. I’m not forcefully fighting against you. And I’m certainly not against Julia as an interactive scripting language. I know how scripting workflows work, I’ve used Matlab for over a decade, and I’m aware of the prevailing mental models of Jupyter notebooks. I don’t even think I’ve been very strong in my posts here — I simply wanted to point out the other side of the coin.

3 Likes

OK, my apologies for misunderstanding.

For sure. I think a lot of people here, myself included from my old C++ days, appreciate the value in not having an exception for interactive use at the top level. You have all done a good job of pointing out the other side of the coin, and for non-interactive/exploratory use-cases, your position is certainly correct.

It sounds to me, though, that if we took @stevengj 's suggestion to make the softscope an option at the script level (and possibly the REPL, though I care less about that) the side effects would be (1) experts would have more difficulty reasoning about scope in their code at the top level (as opposed to inside of functions, which are not effected); (2) static analyzers operating at the top level of scripts (as opposed to inside of functions, which are not effected) would have more difficulty, or might not even be supportable; and (3) Rebugger in its current form is a casualty.

Have I missed key downsides here? In return, we make the experience of new users (and those journeyman like myself who switch between script and function style) much better, and avoid the current state where users run into (non-intuitive to them) scope issues in the first 5 minutes of using the software.

Is there any way to trigger a discussion to move things forward, as it currently seems like all progress has stalled otherwise. Is having an option, where the default of on/off could be discussed, for softscope on the REPL and script level technically difficult to implement? It seems like the least work. Furthermore, the exact softscope implementation used in IJulia has been used for a year without a single issue that I know of.

That was precisely the original intention of this thread, but taking the scoping rules as they are right now, rather than giving ground for yet another debate about whether they should be changed or not. :angel:

1 Like

Rebugger works fine with IJulia, as I understand it, because the softscope transformation doesn’t touch macro contents.

It looks like there is a chance that let’s scoping rule will not be changed (hence not breaking Rebugger): New scope solution - #223 by jeff.bezanson

Speaking of let, how about “wrap for in let together with initialization” as another tip?

While I recognize that occasionally people attribute malicious intentions to those who just disagree with them, from you it is somewhat surprising. I hope that the tone if this discussion recovers from this.

No, I don’t want to “punish” anyone. I just think that even from a pedagogical point of view, sugar-coating the issue for newbies harms them in the long run. Scripting is fine if one mostly does assignments, but nontrivial control flow (which is the most common case of running into the issue, eg loops) belongs in functions. Otherwise, this inevitably leads to “why is my code so slow” questions.

I recognize that you would like to use Julia as a scripting language in teaching where what you call “software development” (which looks like any kind of structured approach beyond trivial scripts) is outside the scope of the course. I understand that it would be convenient for you to do this as you are an expert in Julia. I just don’t think that Julia is well-suited for this, and that we should bend it in this direction.

1 Like

Sorry, I didn’t mean to make you sound malicious.

When you say you don’t want to “sugar-coat the issue” because “nontrivial control flow … belongs in functions,” it sounds like you want working in global scope to be hard as an end in itself. I fundamentally disagree with this viewpoint — we shouldn’t make sacrifices in Julia usability unless it has a clear benefit for production code, which is not the case here (“serious” code in functions is not affected by the scoping rules for global loops).

(I don’t quite understand why you are so passionate about a language rule that, from the sound of it, is irrelevant to the code that you write.)

If we want to make it harder to write slow code in Julia, there are lots of things we could do. For example, we could require a special annotation for type-unstable functions. I don’t think we want to go there.

2 Likes

I think there is a misunderstanding. Doing experiments with created or imported data at REPL is not ‘using Julia as a scripting language’ - actually it’s an interactive way to work. btw: you’ll find “We want it interactive and we want it compiled.” in the " Why We Created Julia".

I work in REPL mostly in matlab, sometimes in python and moving back to an edit-compile-run workstyle would be (yes) moving back in time and experience. Along that i also disagree that fixed length loops (for loops) are a non-trival control flow. In matlab all that implicit indexing stuff (find / logical indices) enable to work without it (when your data has reactangular shape) - while in julia the method of choice seems to be explicit loops.

If julia is not suited to do serious work at REPL, then -well- i’ll move on.

3 Likes

Just to clarify: I think of the combination of nontrivial control flow (for, while, etc) and global state as something that should be possible and has some uses, but I don’t think the language should encourage it as it runs into other issues (mainly performance traps, which are at least equally confusing for newbies).

“Sugar-coating” here refers to the strategy of making a certain kind of coding style easier for newbies, which then, if adopted, runs into inevitable problems later on. I understand that the intention is to help newbies, but I just don’t think it does. This is why I care about this, even though — you are correct — personally I am not much affected as I avoid this kind of code implicitly.

Also, I don’t think that the cost of the current rules is too high, either. It is my impression that people just bump into this once and learn about it, like many other things. But again, it is possible for reasonable people to disagree about this, as we don’t have hard data.

That said, this issue is not black and white for me. I recognize your POV as valid, too, I think we just have a different weight on certain things. Also, I trust the core devs to make the right choice about this eventually. If they adopt a “friendly” version of global scope for scripting that is consistent and easy to reason about, I am fine with that.

We do, however, have the experience of every other dynamic language that is widely used interactively, and none of them behave the way Julia does with respect to scoping of global loops.

(Julia’s design has always been to make as few sacrifices to usability/productivity as possible, compared to other dynamic languages, in order to allow high performance. Not to make sacrifices to try to enforce high performance.)

It’s not just “newbies”, it’s also experienced programmers who want to explore interactively some of the time. Not all code is performance-critical, nor should all code be optimized for performance!

What it comes down to is that the main “benefit” people are citing for the global scoping rules is that it makes commonplace interactive (but slow) code harder to write, not that it makes life significantly easier for “serious” coding. I don’t see this as a compelling argument.

We can’t change the rules now, but we can still improve the interactive situation.

11 Likes

@Tamas_Papp I object to your always repeated assertion that only newbies would ever write loops at global scope. To give some examples:

  1. @benchmark always demands top-level scope (no function!). Hence, lots of ugly annoying global annotations if you want to fill a table with a few 100 benchmark results. Generating + plotting + archiving some data comparing different algorithmic choices and scaling behavior is imo good practice and not “just for newbies”.

  2. Batch processing a bunch of problem instances over night. Of course the functionality is inside proper functions and compiled. But in the end, there is some code-vomit-style script that calls all these optimized functions. Sure you could work to write beautiful scripts, but time is better spent writing beautiful algorithms.

  3. Interactive exploration of data / problems. That’s what jupyter notebooks are super cool for! You want no scoping problems here, and don’t care about perf (because there always is a function boundary before hitting expensive stuff).

Most of my code and workflow is unaffected by global scope questions. But there are literally zero cases where I as a user think “oh, how nice that we have this new scope rule, so much better than 0.6”; instead, it is a constant low-level gripe.

@jlperla gave a perfect explanation of the mental model many people: Top-level code has an enclosing scope, just as every other piece of code, and the same rules apply (for top-level code, the enclosing scope happens to coincide with module-level global scope). This is a perfectly sensible mental model and language design. Julia does something else (also perfectly sensible), where the global-local dichotomy is more pronounced than a “hierarchy of scopes”. I think that everyone here should try to understand both models, not just the one currently chosen in julia.

6 Likes

I think the thread is well-intentioned, but there is a much deeper issue here. The basic consensus among people who do lots of scripting seems to be: (1) Tell beginners, who invariably write script-style code, to start in IJulia; (2) avoid the REPL or julia files for scripting-style entirely and only work with non-script code (i.e. everything in functions). If you tell them that code at the top level simply won’t match their intuition, it will save them confusion until they understand Julia well enough that this isn’t a problem.

I think that making the “tips” more complicated will likely confuse them further. I think having a single, simple tip is the easiest. Explaining a let to someone who doesn’t understand the subtlety of scoping, isn’t going to help.

Please don’t put words in my mouth, I never said this. The pedagogical motivation is prominent for soft global scope though, which is why it was repeatedly used as an argument in favor of it (and I think it is one of the best arguments), and why I used it as an example. This does not mean other uses are not relevant.

I am now going to withdraw from this discussion for a while since I am not comfortable with its tone any more. I understand that there are people who feel passionately about the issue, but I consider it a technical one best discussed in a dispassionate manner. I am afraid that this is becoming very difficult now.

1 Like

I agree that the language would preferably not encourage the use of globals. (Personally I would have preferred all variables to be local, even in the repl or a script, unless explicitly declared global, but no breaking changes allowed …)
I still advocate the use of SoftGlobalScope in the REPL because of the following reasons:

I am convinced that most of the people using things like for loops in the repl do not intend to create and use global variables. Speaking for myself, I have no problem with hard scoping rules in scripts, because I put everything in functions anyway. I do have problems with it in the REPL though: The main advantage of a REPL for me is to be able to interactively test, debug and experiment with code (written in the functions). That copy-pasting code from functions to test creates globals in the REPL is an unfortunate by-effect. The hard scoping rule makes this way of working hard (no longer possible to just copy paste code from functions). To give at least one hard data point, I consider this a very high cost: if not for the softglobalscope solution this might have caused me to abandon further exploring Julia (even though I really like a lot of it).

Interactive use on a repl is also a good way for teaching newbies (in my experience). With soft scope, they can try out loops etc. in the repl and learn about scope when they get at functions, where it is relatively easy to explain. Then you teach them about the performance implications of functions, and tell them to wrap everything into a function. With hard scoping this gets all muddled, having to talk about scoping too soon, and having the simple stuf they did interactively (with the globals in for loops) not work anymore when they wrap it in a function.

For evangelizing about Julia (i.e pitch it to users of other languages), the hard scope also presents a problem. It is different from most (probably all) other languages. There is nothing wrong with being different from others, but you must be able to defend it, to explain why this different behaviour is a good thing. e.g. I’ll happily explain multiple dispatch and its advantages to someone, but I fail to see the advantages to the programmer of hard-scope in this context.

If I had to teach Julia, I’d make sure they get access to a REPL with SoftGlobalScope, and tell them that if they want to run code in a file, they must always put it in a function.
I would like SoftGlobalScope to be the default in the REPL (as it is in IJulia), or at least very easy to use (e.g. command line option).

3 Likes

A post was split to a new topic: Are let blocks as fast as functions?

8 posts were split to a new topic: Scope behaviors in Atom (Juno)

A post was merged into an existing topic: Scope behaviors in Atom (Juno)

made this post in julia usage, juno with atom

it probably should be moved to here since its more about scoping rules in general and clarity of the documentation (wrt scoping and other issues)?

For people (like me) it might be helpful to go for the c/c++ style: if you want to run it “standalone”, it is your main() program. I wrap my script into:

using CoolStuff
include(“important_code.jl”)
function main()
#your script here
end
main()

1 Like