Why is bang(!) indicating argument mutation a convention, but not enforced?

Internals & Design
1

As a Julia beginner/neophyte, I am wondering why using a ! name suffix for functions that may mutate their arguments is merely a convention, but is not enforced by the compiler?

It is nice to have this indication available when using language/package functions, but on the other hand I cannot rely on it (except maybe in the stdlib?), which I think somewhat diminishes its value.
At the same time, if this were strictly enforced, beside increasing clarity of the affected APIs, would that not also afford some avenues for internal optimisations by the compiler?

Iā€™m sure this already came up before, but I searched both the docs and the forum and could not find a relevant discussion/information. :thinking:

2 Likes
2

The only reasonable interpretation would be ā€œno pointer derived from any argument is written toā€.

Consider e.g., map(fun, arr) ā€“ what happens if fun has side-effects that write to memory? What if some methods of fun have side-effects and some donā€™t? Is this call violating a mutability contract? Do you want functions typed by purity?

In short: Neither Juliaā€™s type system nor Julia conventions are a good fit for that kind of stuff. I guess haskell or idris might be languages where this makes more sense (either do some monadic dance or get a compiler error).

Btw, a variety of similar annotations make a lot of sense ā€“ on the method level, not the function level, and as hints to the optimizer (hints as in axioms / assumptions that the optimizer may trust). One of then is @pure, which is almost impossible to safely use, even for core developers (inofficial definition: ā€œa method is @pure if and only if jameson says soā€). Some others are available in llvm.

10 Likes
3

Also, the bang notation is not ideal, it does not indicate: which parameters risk being mutated, if some mutation always occurs or it depends on parameter values; and is, for example, ignored by all the IO functions in Base, because basically everything there would need a bang.

It is a great convention for distinguishing between two alternative functions (one which copy and the other that works inplace) and also as a danger sign for function that are often misused because the users assume they did not change the arguments. But I think trying to make it more than that would take a lot of effort which would not be worth.

8 Likes
4

In this sense, you cannot really rely on anything a package claims to do, because the compiler does not check algorithmic correctness.

15 Likes
5

Thank you for clearing this up for me, this all sounds reasonable! In a way this then seems similar in purpose to Python type annotations - optional, and not checked by the interpreter, mainly meant to ease the work of IDEs/linters/static analysers, and developers.
Iā€™ll think on this some moreā€¦ :slight_smile:

6

It could very well be that, in practice, banning functions whoā€™s name donā€™t end in ā€˜!ā€™ from modifying their arguments is too difficult to be sensible - I donā€™t know enough about Julia to tell (Iā€™m new too). But Tamas_Pappā€™s comment still made me want to express some thoughts.

I cannot rely on comments or (other) documentation being correct. I cannot rely on conventions being followed. I cannot rely on code being bug-free. But, barring bugs in the compiler, I can rely on rules of the language being followed. For instance, if a variable is declared const, I can rely on it not being modified. Or, if a procedure argument in Fortran is declared INTENT(IN), I can rely on it not being modified by the procedure. (In C, declaring a pointer argument to a function as const isnā€™t a guarantee, but gets you pretty close)

If I can declare what some code does or does not do, using language features rather than using comments or conventions, I strongly prefer that. Because comments can be wrong, and conventions can be broken, but language rules cannot. They can be relied upon. Itā€™s nice not just for the user/reader of some code, but also for its writer, as it may catch bugs. What if I really didnā€™t intend to modify a function argument, but by mistake did anyway?

Obviously, there will always be uncertainty about what some unfamiliar code does, or whether your own code have bugs. But while less uncertainty isnā€™t as good as no uncertainty, it is still better than more uncertainty. Letā€™s not refuse to reduce uncertainty just because we canā€™t eliminate it.

If there was a way of declaring ā€œINTENTā€ for function arguments in Julia, be it with exclamation marks at the end of function names or through some other means, I think it would be nice. Not something I canā€™t live without, and if itā€™s decided it isnā€™t worth the trouble, I can accept that, but still - it would be nice.

3 Likes
7

I think you misunderstand what const does ā€” its your promise to the compiler, not the other way round. Cf

julia> const a = 1
1

julia> a = 2
WARNING: redefinition of constant a. This may fail, cause incorrect answers, or produce other errors.
2
8 Likes
8

If someone really cares about this, it would be interesting to see a prototype of a linter that does whole program analysis to verify that a nominally non-mutating function never invokes any mutating functions in any given codebase.

5 Likes
9

In addition to Tamas point that you can misunderstand the documentation, I would like to point out that Julia patch versions exist exactly because the language failed to guarantee its own rules. I myself have already found an unreachable bug in Julia 1.0.5 and had to change versions because the correction would not be back-ported, so the only implementation of the language rules (for some major-minor version) may be broken and without prospect of repairs.

1 Like
10

Oh? It seems Iā€™ve learned something new today, then - thanks! :slight_smile:

But let me point out that it still gives a warning. If you would declare that a is meant to be a constant only through some naming convention or a comment, you would get neither errors nor warnings when itā€™s modified.
Itā€™s not an unimportant difference.

1 Like
11

While the request here might not be enforceable, perhaps a useful extension would be for the function signature to document which arguments are mutable as well?

Something like

function push!(collection!, item)
    # do stuff
end

a = [1,2]
push!(a!, 3)

Though I guess that would require changing ! from just a character that can be part of a symbol to an optional decorator, which would break things.

12

But hereā€™s a discussion on a related topic that is helpful: https://github.com/JuliaLang/julia/issues/26484

2 Likes
13

No one is arguing that. But a feature like this a lot of work and will probably be implemented only when someone wants it bad enough to invest the work.

While using ! in function names is not a 100% perfect solution, it is good enough for in practice for a lot of things.

1 Like
14

Thank you all for the interesting inputs and perspective. And especially @ErikE for formulating my intention in a much more eloquent way - I am more or less 100% aligned with what you say, about language features like INTENT, or const (at least the non-Julia one), or Modelicaā€™s constant/parameter/variable variability indication, language features vs. convention vs. comments, etc. (btw, I was also surprised about the ā€œdirectionā€ of the const-promise :smiley: - thanks!)

I am wondering - what mechanism triggers these warnings when you modify a const object, and how well does that work (is there a limit to the detection heuristic)?
Canā€™t this same mechanism be adapted to warn if your function-without-bang mutates any of its arguments? After all, doesnā€™t a function-without-bang essentially correspond to having only const arguments (at least in the functionā€™s scope)? Could that be leveraged somehow? E.g. could the compiler just slap consts in front of all function arguments if there is no bang at the end of the function name? (Surely itā€™s not as easy as that)

15

Noā€“it really doesnā€™t work this way. The warning happens when you assign a new value to a variable which has been marked as const. Thatā€™s completely different than mutating an existing value, which is what the ! suffix refers to.

Try this:

julia> const x = [1, 2, 3]
3-element Array{Int64,1}:
 1
 2
 3

julia> x[1] = 10
10

x has been mutated, but there is no warning, and the resulting code will work correctly with no issues. Thatā€™s exactly as intended: the const label has nothing to do with mutation of an existing value. Instead, the warning occurs if you assign a new value:

julia> x = [4, 5, 6]
WARNING: redefinition of constant x. This may fail, cause incorrect answers, or produce other errors.

No, definitely not, and for the same reason. A function with ! may mutate its arguments (which is not something const cares about).

As an exercise, consider:

function f1(x)
  x = [1, 2, 3]
end

and

function f2!(x)
  x[1] = 1
  x[2] = 2
  x[3] = 3
end

Both are perfectly legal functions, but only f2 will modify the value which was passed into it. f1 just creates a brand-new value with the same name in its local scope and has no effect on the value you pass in.

5 Likes
16

Perfectly clear, thanks for indulging this greenhorn! :slight_smile:

1 Like