I am curious to know how much people want to have a feature/mechanism in Julia to manually control function overloading. Specifically, to disallow overloading already defined methods when intended.
One direct motivation for such a restriction is to prevent malicious/unwanted code injection. Consider the following example:
julia> import Base: + # First line after a freshly started Julia REPL
julia> +(::Int, ::Int) = zero(Int) # DANGEROUS!!!
Executing the second line of code will cause the Julia process to crash immediately.
As far as I know, there is no prevention from someone overloading an already defined method, which, in a worst-case scenario, can lead to the consequences of the above example.
Of course, one can argue that it is not Julia’s “responsibility” to prevent the end user from exploiting the vulnerability of multiple dispatch. In my opinion, the discussion about a language’s responsibility (e.g., restriction vs. freedom) for its users is fundamentally a philosophical topic in language design, which I shall refrain from further exploring. Instead, I would like to discuss my concerns and thoughts more concretely in relation to some more subtle scenarios and potential implementations.
Prevent silent function misuse and (some) type piracy issues
What if, when someone overloads a method, their modification does not crash the program but causes silent wrong/unintended behavior for other callers (e.g., functions) of that updated method? In my opinion, this happens much more easily when the user overloads a method from a library (module) not created by them. In other words, such behavior can be categorized as a special case of type piracy: defining a method f with respect to a set of argument types {T}, where neither f nor {T} is owned by the user.
Admittedly, there can be some convenient use cases to “justify” such a “piracy”. However, in my opinion, overloading already existing methods that the user does not own is almost certainly a bad practice, as it potentially introduces performance (compilation invalidation) or correctness (silent incorrect results) risks. As far as I know, currently, there is no formal way to prevent type piracy. Having manual control over whether a method can be overloaded can be a good indicator for the library developer to warn the user about which parts of the library are not encouraged to be modified or extended. One dilemma I sometimes encounter as a user is when I want to overload a function from a library, yet I’m unsure if it will cause unwanted side effects, as I don’t fully understand the source code like the developers. This risk is especially amplified when the user is also a developer for another library. Many Julia libraries attempt to depend on other upstream libraries by overloading public but non-exported functions. Therefore, I believe, it would be beneficial for the language and the entire Julia ecosystem to provide an opt-in feature that prevents further overloading of existing methods that “are not meant to be overloaded”.
Potential syntax proposal
I’m not sure whether such a feature can be realized by a macro or an entirely new keyword. For the purpose of demonstrating how syntactically this feature might work, I will use a macro “@protect” to illustrate. Please feel free to propose alternatives.
The basic syntax for a concrete method protected from further overloading can be:
@protect function foo1(a::Int, b::Int)
a + b
end
Once foo1 is compiled (either through precompilation or JIT compilation) at a world age, it cannot be overloaded afterward with a new function body. This is the most basic case, but things can get a bit more interesting once we allow abstract types to be in the function argument signatures:
@protect function foo2(a::Real, b::Int)
a + b
end
In this case, any new method foo2(a::T, b::Int) where T is a subtype of Real is disallowed to be overloaded. In other words, the function signatures with abstract types enclose a set of concrete methods that are all forbidden to be overloaded in the future. In an extreme case:
@protect function foo3(a, b=1)
a + b
end
All methods of foo3 with one or two arguments shall not be allowed to overload. I haven’t thought about how to incorporate (optional) keyword arguments into this syntax, since they are not supposed to participate in multiple dispatch, even though there are lingering complications of its practical behaviors. At a minimum, we can disallow keyword arguments in this syntax.
Enable (potentially) more code optimization
I will preface by saying that I don’t know much about the low-level (on the IR or LLVM level) optimization of Julia code, so please correct me if I am wrong. And please point out if there are any more potential benefits!
Based on my understanding of multiple dispatch, a global method table is created to store all specialized methods when a Julia process starts. If some methods can be marked as “cannot be overloaded”, these methods can potentially be stored in a separate table for more specialized optimization, e.g., inlining methods. Furthermore, this feature may also help with type inference, as it can implicitly preserve more information about methods defined by abstract-type arguments (note that the function body of foo3 is fixed).
Compatibility
Since this is proposed as an opt-in feature, it is disabled by default unless the syntax is actively used. Hence, I think it can be added to Julia without introducing breaking changes, assuming it is realizable in principle.
This is overall a rough sketch of how I imagine we could potentially have better control over method overloading (and underlying multiple dispatch) in Julia. Regardless of whether it is necessary or practical, I would like to use this thread to initiate a discussion within the community. Any related comments are welcome!
Thank you for your time!