Clarify the documentation about modifying module variables

[benibilme]

There is weird description without any example in the documentation.

Julia 1.10 Global-Scope definition

Note that while variable bindings can be read externally, they can only be changed within the module to which they belong. As an escape hatch, you can always evaluate code inside that module to modify a variable; this guarantees, in particular, that module bindings cannot be modified externally by code that never calls eval.

What does that mean? What are the mechanics? Why need an escape hatch?

[goerz]

I was going to say it means that the following does not work:

julia> module A

       data = 1

       function set_data(value)
           global data
           data = value
           return data
       end

       end
Main.A

julia> using .A

julia> A.data
1

julia> A.data = 2
2

Alas, it does work, which is very confusing (and maybe a bug???). I’m pretty sure that if that module was in a proper package, it wouldn’t work. Nope, still works, which contradicts what I thought I knew about Julia. So, thanks for asking the question!

Still, based on my current understanding, the “recommended” way of changing module data would be to use a function from inside the module, which can modify the data:

julia> A.set_data(3)
3

julia> A.data
3

And the “escape hatch”, if no such function is provided, is to use

julia> A.eval(:(data = 4))
4

Take this clarification with a grain of salt, since I seem to have some misconception here. I agree that the example given in the documentation itself is not very illuminating.

[Mason]

@goerz

Alas, it does work, which is very confusing (and maybe a bug???)

Not a bug, so much as I think everyone forgot that section of the manual existed when implement `setproperty!` for modules by simeonschaub · Pull Request #44231 · JuliaLang/julia · GitHub was written. So I think the docs just ought to be updated.

[goerz]

Huh. I totally missed this change in Julia 1.9!

I’m also not sure how I feel about it. I’ve been relying on the old behavior to ensure module data can’t easily be set to an invalid value.

In any case, the documentation should definitely be updated:

Remove outdated discussion about externally changing module bindings

[Mason]

It doesnt let you do anything you couldnt do before using eval so I dont think it’s really much to worry about (it’s a strict subset of evals capabilities).

It’s just a special case that’s easy for us to optimize in a way that we cant optimize a fully general eval.

[Benny]

@goerz

I’m also not sure how I feel about it. I’ve been relying on the old behavior to ensure module data can’t easily be set to an invalid value .

I was also under the impression that reassigning globals from another module was just too dangerous to allow dot syntax until v1.9 came along and implemented it. We just have to use the typical practices instead: 1) making the variable non-public (this will be more formal with a public keyword in an upcoming version) and only exposing the getters and setters as public, 2) when possible, consting the variable so even bad-faith users can’t reassign it.

[benibilme]

I still do not understand what is going on. Could someone interpret it for me? Does it mean that the global variables can be accessed with a . notatiton such as A.a?

[goerz]

It doesnt let you do anything you couldnt do before using eval

Not if you’re determined, but an unsuspecting user is very likely to just set the (compltely invalid)

QuantumControlBase.DEFAULT_AD_FRAMEWORK = "Zygote"

as opposed to using the set_default_ad_framework setter that ensures that Zygote is actually loaded and isn’t confused about the difference between a symbol and a string.

On the other hand, if they decide to use eval, I’d hope they understand that they’re leaving behind all pretenses of an official API, and any resulting breakage is on them.

[Benny]

“calls eval” is just an off-hand mention, it doesn’t serve as an example like A.eval(:(data = 4)), as goerz said earlier, or @eval A data = 4. You’d have to look up eval to figure out these, so it’s normal not to immediately understand when you’re unfamiliar. Some passages are just dense with background information that reading comprehension slows to a crawl for anyone.

@benibilme

Does it mean that the global variables can be accessed with a . notatiton such as A.a?

Yep, you can test these yourself in the REPL.

julia> module A data=1 end
Main.A

julia> A.data=2
2

julia> A.data
2

The writing that says you can’t do this and must instead use eval, has been outdated since v1.9.

[mnemnion]

@Benny

We just have to use the typical practices instead: 1) making the variable non-public (this will be more formal with a public keyword in an upcoming version) and only exposing the getters and setters as public, 2) when possible, consting the variable so even bad-faith users can’t reassign it.

I would argue that constants should be declared with const, period. Relying on privacy to preserve an invariant is just security through obscurity, and it isn’t just users of the package which can modify a non-constant global. The compiler has to assume that the value can change at any time, which is inefficient. It’s only arguably “bad behavior” for a package user to change a variable, it’s variable! If the semantics don’t allow it to vary, don’t make it variable!

A package with global state which is expected to only be modified internally has a strong code smell. It means that the package can only be in one of those states at a time, and it means that whatever internal function modifies that state modifies it for all users of the package, including ones which were expecting it to be in state A, because it was, before something else put it in state B. Using dot-syntax to mutate that state directly is just a special case of this problem.

This isn’t a moral argument, one of my packages has some global state in the form of a (constant) Dict mapping symbols to behaviors. But I’ve taken pains to make sure that any unauthorized mutation of this Dict won’t break anything compiled against earlier, correct versions of it.

I suspect Julia will eventually add a private keyword which makes it arbitrarily difficult (if not impossible) to modify variables so marked from outside the package. If it’s important, one can use a gensym right now to make it very clear that something isn’t to be messed with. But none of these things will change the fact that global mutable state is nearly always a bad idea, especially if it’s load-bearing.

[Benny]

Sure, keep invariants constant or immutable, but not everything is invariant, goerz provided a reasonable example. Julia lacks actual access modifiers in general, and my understanding is this gets out of people’s way when they need unrestrained access to implement controlled access. If someone breaks an instantiated module because they changed an internal unsafely against the public API, then that’s on them. Julia doesn’t stop you from breaking itself in other ways either, like adding or redefining Base methods on Base types. I would agree on the gensym, though that doesn’t stop reassignment and mutation either and I tend to just do underscore-led names. The upcoming public keyword does not seem to attempt to modify access, and I doubt a derivative private would do so either.

Also, could Base.ImmutableDict help your use case? You would have to specify all entries at instantiation, and if you need to progressively collect entries first, you could do that in a local scope with variables that go away.

[mnemnion]

Yes, the converse applies: if something is genuinely variable, make it a variable. At which point it’s certainly not bad behavior for a user to modify it, that’s expected behavior, and a dot-syntax on modules is coherent with the rest of the language.

What isn’t good is mutable global state which isn’t resilient to mutation from outside the module. I’m trying to avoid being doctrinaire here, but that combination very often means that changing that global state will modify behavior from “upstream” of the change, and that’s a bad problem to have. The user setting that global to the wrong value is merely a special case of the problems which might arise.

public and private don’t have to carry the same semantics they carry in languages which get them from C++. Julia’s upcoming public is documentation, as I understand it, a way of marking something explicitly as part of the API without exporting it. It wouldn’t be needed if Julia didn’t have the using keyword, but it does, and in my opinion the convenience of using makes up for the problems it can cause.

private, if it’s ever added, can serve to prevent access to a variable outside of the defining module. The compiler could use that directive to eliminate the symbol entirely, so there’s no variable left to access, just compiled references to the memory it’s using. Whether this is philosophically compatible with Julia isn’t something I have an opinion on, but it’s definitely possible to add a feature like that.

I wish ImmutableDict were called something else, actually, like AList, because that’s what they are: a linked list of key/value pairs. Plenty of times that such a thing is useful, but it isn’t useful for my case, because adding a second definition of a key shadows the first. What I’d like is a Dict where redefining a key has no effect, Base doesn’t have one of those, and it’s not quite worth writing my own wrapper struct over Dict to provide that behavior. At the end of the day, it’s the user’s code once they load it, if they want to break it, I’m not strongly motivated to prevent them from doing so.

I would make it private as well as const if it were possible to do that, fwiw. it’s intended to be an append-only cache, which is only added to using a particular function, and if I could enforce that as an invariant, I would, it’s good practice.

[Benny]

@mnemnion

Julia’s upcoming public is documentation, as I understand it, a way of marking something explicitly as part of the API without exporting it. It wouldn’t be needed if Julia didn’t have the using keyword

Traditionally, separate documentation was considered enough to inform what was public vs internal, but enough people felt this was unsatisfactory and should be indicated directly in the source code and discoverable by reflection functions. Whether a name was affected by using wasn’t much important; export lists are widely understood to be only a subset of public API.

@mnemnion

no variable left to access, just compiled references to the memory it’s using. Whether this is philosophically compatible with Julia isn’t something I have an opinion on, but it’s definitely possible to add a feature like that.

My understanding is that it isn’t, otherwise v0 would have provided true access modifiers for the much more widespread fields in addition to globals. Worth mentioning for OP that getproperty and setproperty! can be defined for a type to prevent dot syntax, but that doesn’t stop the core getfield and setfield!, or for modules getglobal and setglobal!. The implementation of dot syntax for global variables in modules seems to recognize that abusing dot syntax is generally just as bad as abusing those core functions or eval, so it’s unreasonable to gatekeep dot syntax only for modules.

It’s actually already possible to make a reference without a variable by interpolating instances into an expression, like @eval getcache() = $(Dict()). But that doesn’t generally stop bad-faith access because if any method returns the interpolated mutable instance, you can mutate it. That topic is getting away from the thread topic so I’d suggest making a new thread or direct message if this continues further.

[mnemnion]

@Benny

Traditionally, separate documentation was considered enough to inform what was public vs internal, but enough people felt this was unsatisfactory and should be indicated directly in the source code and discoverable by reflection functions. Whether a name was affected by using wasn’t much important; export lists are widely understood to be only a subset of public API.

To illustrate the connection, consider a counterfactual Julia in which using does not exist. Instead, names must be imported explicitly using import. Furthermore, only names which are on the export list can be imported: other names must be qualified with the module name.

This would prevent the issue which using poses, which is that adding a name to the export list can clash with names in modules which use the package, so a minor update can break downstream code, even without violating SemVer. This wouldn’t change the semantics of Mirror World Julia, all names from a module may still be accessed, they can be localized with const foo = TheModule.foo and what have you, but no global importing with using, and import only works on export.

Such a language would have no use for public, because the export list is the evident public API. You can’t break anyone else’s code by adding names to the export list, because using doesn’t exist, so neither do implicit imports. The motive of public in actual Julia is that people actually use using, so there’s a desire to limit the export list to only the core features of a package, and have a second list of names which are officially supported in the API but aren’t imported automatically.

using is great for the REPL and the exploratory period of writing code, but it’s good practice to replace it with an explicit import list once things are settling into shape.

It’s quite tractable to write module code in which values intended to be private aren’t returned by any of the methods which use them, so if there were to be a private keyword, it would in practice protect those values from mutation outside of the module. I understand why a language might not want to provide this feature, but would make limited use of it were it present. I’ve dealt with the annoyance of deciding whether to modify some dependency to un-private a part of a class which I’d like access to, and for the most part, I don’t see the point in preventing people from burning their own fingers. I’ve also worked on codebases large enough to appreciate the ability to make it impossible to access certain state without explicitly modifying the code to do so.

[benibilme]

According to documentation, above code must be wrong as stated below. However, I tried it and it works. Possibly another out-of-date section in the documentation?

Julia 1.10 module documentation Qualified Names Section

Within a module, a variable name can be “reserved” without assigning to it by declaring it as global x. This prevents name conflicts for globals initialized after load time. The syntax M.x = y does not work to assign a global in another module; global assignment is always module-local.

[Benny]

@benibilme

The syntax M.x = y does not work to assign a global in another module; global assignment is always module-local.

Yes this sentence is outdated since v1.9