Naming of ScopedValues

Internals & Design
1

A bit off topic, but it confuses me that my old friend from Lisp, the *dynamic-variable*, is called ScopedValue in Julia. I gather that’s what they’re called in Java?

My gripe is that what differs between lexical scope and dynamic scope isn’t scope, but lexical vs. dynamic. So it’s an odd choice to elide DynamicallyScopedValue as ScopedValue, and not DynamicValue.

Ah well. A feature existing is more important than what it’s called, after all. I’ll just file it under “naming things is one of the hard problems” and move along.

4 Likes
2

The original PR linked above (Scoped values by vchuravy · Pull Request #50958 · JuliaLang/julia · GitHub), has an extensive analysis and bikeshedding of the naming. Suffice it to say all of those were considered, but ultimately the consensus for ScopedValue was the most robust. This is not a case of people pulling a name out of nowhere without consulting all the prior art.

4 Likes
3

Yeah, reading that was how I knew that the terminology was borrowed from Java, which I will note is not renowned as a wellspring of excellent names. I expect the choice to lead to an eternity of conversations which go like this: “What’s a ScopedValue?” “It’s a dynamic variable” “Why is it called that, don’t all values have scope” “Don’t know, that’s just what Julia calls them”

Relatively harmless in the grand scheme of things, but I don’t at all consider “all values are scoped, what makes this one special is that the scope is dynamic” to be a bikeshed. Reading that thread I don’t get the impression of a thorough decision based on the “robustness” of the chosen term, whatever that means, just no one willing to go to bat and spell out why it’s a bad name. I agree with this comment with the refinement that DynamicValue is an obviously better name, the decision was hastily made.

Certainly not worth endlessly relitigating or attempting to change it, just too bad, is all.

5 Likes
4

The precise reason I mentioned the length of the thread was to disabuse anyone of the notion that this was some kind of shotgun naming decision. I know it’s a pain to expand long GitHub comment threads, but once expanded one can clearly see comments like Scoped values by vchuravy · Pull Request #50958 · JuliaLang/julia · GitHub which have carefully considered stuff like lisp heritage and whether naming decisions from it still make sense (with a good motivating quote from Guy Steele, no less).

FWIW, there were even longer debates on Slack around naming. Like all things Slack though, those are no lost to history.

1 Like
5

My disagreement with the decision is pretty mild and I’m afraid I’ve taken this off-topic, but since you brought that post into scope, as it were, I feel I must ask: if dynamic scope is a misnomer, why ScopedValue? Surely the dynamic nature of the construct is not in question?

They also aren’t rebound on the basis of scope, you can’t use let to rebind a ScopedValue the way you can in Common Lisp, instead this has introduced a do-notation function with which provides the appearance of scope, when (and only when) used with do-notation. The disconnect is fairly complete, in fact.

It distresses me that the community continues to use Slack for any matters of substance. That’s a bad habit.

But really I’d prefer to drop it, I’m not in the habit of fighting losing battles, let alone ones which are already lost.

6

A much more likely flow of conversation (in these parts) would be

“what’s a ScopedValue?”

“It’s a dynamic variable.”

“What’s a dynamic variable?”

I’m not here to defend “scoped value”, but “dynamic variable” is also a crap name in its own right, and the learning resources that explain what it is are extremely unhelpful to your median julia programmer.

1 Like
7

I would prefer DynamicValue over DynamicVariable, the latter really should refer to a variable which is dynamically bound to an instance of any value or type, so in Julia, just a an ordinary variable, without const or a typeassert, each of which makes it static in a different way.

But the terms lexical and dynamic scope are canonical and have been for some decades, and I don’t see how one explains what a ScopedValue is without reference to that distinction. The first sentence of the upcoming documentation is Scoped values provide an implementation of dynamic scoping in Julia, there’s really no getting around it.

“So scope means dynamic scope?” “No, scope is scope, Julia uses lexical scope, and ScopedValues for dynamic scope”.

Then you get to explain that the dynamic scope doesn’t use the scope mechanism, but rather a special-cased function which introduces a dynamic bind within its scope. The word “scoped” in ScopedValue clearly comes from “dynamic scope”, and doesn’t mean what “scope” means in the entire rest of the language. It’s an odd choice.

1 Like
8

DyanmicValue is a pretty meaningless and confusing name though, because all values in julia are “dynamic” since julia is a dynamically typed language.

Besides that, ScopedValues aren’t variables. They’re containers whose de-referenced value depends on dynamic scoping information. If they were variables, then I’d say we should probably call them dynamic variables, but they’re not variables.

9

It’s a heritable task-local variable, together with some syntactic sugar to set/restore it.

For me this has nothing to do with scope, except that java called its equivalent construction also ScopedValue :wink:

The magic parts are the HAMT, and the fact that task creation shallow-copies the HAMT. Exactly what’s needed for e.g. logging contexts.

2 Likes
10

Perhaps it should have been spelled like this:

x = DynamicallyScopedValue(1)
f() = @show x[]
dynamic_scope(x=>5) do
    f() # 5
end
f() # 1

…so, DynamicallyScopedValue and dynamic_scope instead of ScopedValue and with. Sometimes clarity calls for more verbosity.

11

I have zero intention of re-litigating the discussion which already happened around naming :slight_smile: . My only purpose in directing people’s attention towards the GH discussion is to ensure there aren’t any misunderstandings caused by insufficient context on what was considered at the time. For example, DynamicallyScopedValue and the verbosity thereof were already discussed on the issue thread. As long as nobody is repeating arguments from that thread believing they’re completely novel, I’m probably going to mute this topic and let any lively follow-up discussion run its course.

3 Likes
12

In certain contexts we refer to rvalues and lvalues, but in general (and this is how the Julia documentation uses these words) lvalues are called variables, and rvalues are called values.

Values (rvalues) in Julia have types. This is true of any language higher-level than Forth. It is also possible to give lvalues types, but of course we agree that the lvalues are dynamic, since without annotation, you can change both the value and type of a variable. I could quibble that “really” it’s a default top type of Any, but Julia refers to itself as a dynamic language and I see it that way also. This doesn’t seem like the place for a philosophical digression about the taxonomy of a language with a fully-integrated gradual typing system, but the fact that type errors occur at runtime is fairly conclusive, if we’re using a binary of static or dynamic, imho.

The rvalues are not dynamic. Some are mutable, some immutable, but without resorting to reinterpret they are of a single static type for their lifetime.

ScopedValues give you a value (rather than a variable) which is dynamic, specifically, dynamic in scope. Observe:

julia> a = ScopedValue(1)
ScopedValue{Int64}(1)

julia> b = a
ScopedValue{Int64}(1)

julia> with(a => 3) do; println(b) end
ScopedValue{Int64}(3)

This is not a static value, it is a dynamic value. I would argue, as long as we’re doing language taxonomy, that this is true of Refs as well. But they aren’t scoped.

Which is why, despite having spun off a whole thread to kibbitz about it, I don’t especially care that ScopedValues was chosen for the name here. I hadn’t booted up nightly to play with the implementation, and after doing so I would even say that ScopedRef might be the best choice, that’s pretty much what they are. I had assumed before reading the manual that access would look like a not a[], which feels less like a classic dynamic variable. So be it.

It’s a decent feature to ship, easy to implement, minimal changes to the language itself. That does leave some rough edges, as the manual warns “Scoped values are constant throughout a scope, but you can store mutable state in a scoped value. Just keep in mind that the usual caveats for global variables apply in the context of concurrent programming.” This implies to me that if you change the binding of a ScopedValue with a with closure, and some other Task in, ahem, a different scope, happens to read it while the other thread is inside the closure, it will get the inner value, not the outer one. Which is unsatisfactory, and fights against the “scoped” part of the concept (not talking about the name for once!).

I’m not convinced that first-class dynamic values (or dynamic variables, as they’re called in most cases) are worth the language complexity they would bring. Those would work like this:


dyn = Dynamic(0)

function closed(i)
     global dyn + i
end

Threads.@thread for i = 1:10
    let global dyn = rand(1,10) # This is thread-specific
        println(closed(i))
    end
end

The let-bound value of dyn (which is accessed without []) would be correctly bound by the scope, in each thread, to the result of rand, within closed. Because this hypothetical construct uses assignment to rebind the value, a const dynamic wouldn’t allow rebinding (and its dynamic nature would therefore be wasted). Although really the interaction with const and typeasserts could be whatever the implementation wanted, because this would have be baked in on a fairly deep level. There are several subtle questions I’m not even going to try to answer here.

Making this work would call for far-reaching changes to the compiler, and I am not at all convinced that anything like this is worth having. If (I haven’t confirmed this) the ScopedValues implementation isn’t thread-safe, that would be a good thing to change.

I’d really like to wind this down because I’m sure it gives the impression that I care a lot more about the name in question than I actually do. I want to conclude by pointing out that my issue with ScopedValues is that they do not use the scoping mechanism, rather than the truly minor quibble that the dynamic part of the term “dynamic scope” should have been chosen. The reason that matters is that what’s been implemented here is the dynamic bind part of how dynamic variables work in Lisps. Not the scope part.

If they can manage to make them thread safe, I will no longer care at all. Show me a language which has no weird or questionable names for elements of the language. You can’t.

13

A Ref is always mutable and ScopedValue is immutable.

No this is a misunderstanding. The problem with putting a mutable value into a ScopedValue is one of data-races. If within the same dynamic scope two programs are writing to mutable the mutable content of a ScopedValue you can observe data-races. A mutable scoped value is something like ScopedValue(Ref(0)).

In your example the outer scope will always get the outer value and the inner scope will get the inner value, but you may see concurrency races if they share data.

ScopedValue are thread-safe, their contents may not be.

2 Likes
14

Immutability as a concept gets a bit hazy when we start talking about dynamic binding, but sure, I see what you mean. “A Ref which can only be mutated using a specific sort of scope-like construct” could be called a ScopedRef, perhaps. Or a ScopedValue. That’s fine too.

I don’t 100% understand what you mean here, but if it means that the user can do different dynamic binds in different threads, that’s good. I don’t see how “usual caveats for global variables apply in the context of concurrent programming” is applicable if that’s the case, but that’s just an issue with the documentation.

Taking a closer look at it, I can see that this sentence was intended to refer to mutable state being referenced by a ScopedValue. I think this could be tweaked to be a bit clearer.

As a specific suggestion, the example above that sentence could spawn threads and then do dynamic binds. Putting the threads inside the bind gave me the wrong impression of what the following sentence was getting at.

15

The ScopedValue name and the semantics of with and @with and => is wicked ugly. Why not just dynamically “scope” the function by re-defining it everywhere it is to be dynamically scoped?

macro dyno(exs...)  # not robust to types and stuff, just demo
   esc(Expr(:block, map(exs) do ex
      if ex isa Symbol
         typeof(getfield(__module__, ex))()
      elseif ex.head === :(=) || ex.head === :function
         :($ex; (::Type{typeof($(ex.args[1].args[1]))})() = $(QuoteNode(ex)))
      else throw("idk wut fk ok")
      end
   end...))
end


@dyno f() = a
@dyno g() = b
@dyno h() = c
a, b, c = 1, 2, 3
f() + g() + h()  # 6

let a=2, b=3, c=4
    @dyno f g h
    f() + g() + h()
end  # 9
16

Do you really want to dynamically set values for all the names currently in the caller’s scope? That could be hundreds of names in a big file.

17

While this works in this small example, I think it would not be practical for multiple nested calls to functions using dynamic scope. Essentially you have inline the full call stack. This gets even worse when functions have multiple methods because you’d need to inline all of them.

Another reason why it’s not that easy is that this method messes up the normal local scoping. With this inlining you make every variable dynamically scoped with no chance of distinguishing scoping between variables within a function.

3 Likes
18

Ah, I see what the problematic use case is: when you want to call a whole stack with a deeply nested dynamically-scoped callee (maybe it’s a low-level routine that relies on a “global”-ish setting, like a numeric tolerance or an iteration depth), but don’t wish to dynamically scope the rest of it.

Nonetheless the point about naming and semantics stands, it’s a wart on the language. These are probably best thought of as “dynamic constants.” Could we have a contextual keyword like dynamic for this, as @mnemnion suggests? Something like this seems far more appealing:

dynamic const a = 1
f() = a

f()  # 1

dynamic let a=2
    f()
end  # 2
19

your example also doesn’t work for the most important part of this: thread safety. A ScopedValue will behave correctly in the face of concurrent uses of with.

That’s a different feature. That’s a dynamic binding, whereas ScopedValue is a container. Feel free to open a PR implementing dynamic bindings using the existing ScopedValue infrastructure though

20

The solution offered by ScopedValues is hard to beat for Julia 1.0. It adds a new type which is accessed like a Ref, and it adds a new do-notation function which pretends to be a scope. None of the internals change, but you get dynamic bind out of it. You pretend that a => b is an a = b and that a with() do ... end block is a scope, and the code gets written. I’ve said more than I need to about the name, but the implementation seems basically sound, especially given that I was wrong about its thread-safety.

It’s not clear that dynamic bind is important enough to integrate any deeper than this. The changes would touch the whole compiler, and I wouldn’t even say the result would be an improvement. There’s a reason why Common Lisp style puts earmuffs on dynamic variables, you want to keep track of them or things get confusing. The a[] access pattern and the with “scope” block play the role of *earmuffs* here, informing readers and writers alike that lexical rules aren’t being followed.

Dynamic bind isn’t something I’ve missed in languages which don’t offer it. I might find myself reaching for a ScopedValue someday, but then again, maybe not. It’s a nice-to-have but I wouldn’t say it’s worth baking in any deeper.

4 Likes