Weird overwriting/extension of an operator based on whether it was ran beforehand

New to Julia
1

I was reading up on automatic differentiation, and I noticed something weird about extending operators. I opened up the REPL and ran a couple of copied lines:

julia> struct Dual{T<:Real} <: Real
x::T
ϵ::T
end

julia> a::Dual + b::Dual = Dual(a.x + b.x, a.ϵ + b.ϵ)

  • (generic function with 1 method)

Wait a minute, I wasn’t supposed to be able to extend an existing operator just like that. And addition (+) is definitely not 1 method. I try adding two integers, and it looks like I’ve overwritten the operator in a bad way:

julia> 3+5
ERROR: MethodError: no method matching +(::Int64, ::Int64)
You may have intended to import Base.:+
Stacktrace:
[1] top-level scope at REPL[3]:1

I close and reopen the REPL: fresh start. I check the + operator first thing before running the copied lines. Now Julia throws the error I expected when extending operators without explicitly using Base, and I actually do add 1 method to Base.:+ when I do it right:

julia> +

  • (generic function with 166 methods)

julia> struct Dual{T<:Real} <: Real
x::T
ϵ::T
end

julia> a::Dual + b::Dual = Dual(a.x + b.x, a.ϵ + b.ϵ)
ERROR: error in method definition: function Base.+ must be explicitly imported to be extended
Stacktrace:
[1] top-level scope at none:0
[2] top-level scope at REPL[2]:1

julia> Base.:+(a::Dual, b::Dual) = Dual(a.x + b.x, a.ϵ + b.ϵ)

julia> +

  • (generic function with 167 methods)

Of course, since I didn’t check the subtraction (-) operator, the weird overwriting still happens for it.

julia> a::Dual - b::Dual = Dual(a.x - b.x, a.ϵ - b.ϵ)

  • (generic function with 1 method)

What exactly is going on here? Is there a way I can make Julia “check” all the operators so I don’t overwrite an operator in a session and have to start over?

2

One note is that if you keep your work in a file, it removes the need to start over. Just re-run the file

3

yes that’s how I usually do things but sometimes I just want to paste a bit of code once and see how it works. I usually don’t run into this problem because people usually give methods new names instead of extending operators

4

Another part of the answer is that if you know you will be over-rising methods, you should always just import them first. It just makes life easier.

1 Like
5

Let’s look at the following session, in a fresh Julia REPL, and try to explain what goes on at each step:

# (1)
julia> foo(x) = 2 * x
foo (generic function with 1 method)

julia> foo(1)
2

#(2)
julia> +(a,b) = 42
+ (generic function with 1 method)

julia> 2 + 3
42

# (3)
julia> *(a,b) = 42
ERROR: error in method definition: function Base.* must be explicitly imported to be extended
Stacktrace:
 [1] top-level scope at none:0
 [2] top-level scope at REPL[5]:1

julia> 2 * 3
6

step (1)

This should not be surprising to you: you create a new function named foo. Now it so happens that, as of now, no function foo actually exists in Base, so that there is no ambiguity about what foo(1) refers to.

But what if Julia v1.5 introduces a new function foo in Base? If foo(1) is made to automatically refer to Base.foo following that change, it will mean that this change in Julia v1.5 is breaking your code. Potententially, this means that the introduction of any new function into Base (or any package that you use) is a breaking change, which we do not want.

Instead, since the first use of foo is its definition, Julia will always consider that you’re trying to define a new function foo for later use in your program, regardless of whether Base.foo exists (now or in the future)

step (2)

For the reason stated above, this is introducing a new function that happens to be named +. And all uses of + in your module will refer to this. But you haven’t really broken the regular Base.+: it is just shadowed by your local implementation. All other modules/packages still use the regular implementation of + that they can find in Base.

step (3)

When you try to define *(a,b), Julia notices that you’re already aware of the existence of a function named *: indeed, you’ve just used it when calling foo(1)!
You wouldn’t want the * in foo to refer to Base.*, when it would refer to a newly created function everywhere else in your module. So Julia errors out, saying that you probably want to create a new method instead, extending Base.*.

And such an extension should be explicit, because it has the potential to break everyone else’s code (for example if you commit type piracy).

Is that clearer?

8 Likes
6

Good points here about preventing changes to Base from breaking existing code and vice versa. I’d much prefer dealing with this minor annoyance than with multiple dispatch choosing unintended methods just because of a name conflict between my code and Base.
Something to point out though: the example in my question is a counterexample where not referring to Base.:+ is what breaks it. I didn’t run addition of Duals in the first session of my post, but if I had, it would’ve thrown an error because the (+) operator no longer knew how to add Reals. It’s easy to change the method declaration from “blah + plah” to “Base.:+(blah, plah)” or, what I’m learning might be a far easier option, a different name entirely.

7

Yes, exactly.

So when you create new functions when what you really mean would be to extend an existing function by adding new methods, you code is likely to fail, one way or the other:

  • either you use the function from Base before defining you new function, and Julia errors out with a clear error message,
  • or you define your new function first, and your code breaks as soon as you need a method of the pre-existing function from Base.

I’d argue that this is more on the “feature” side, because it helps you preventing errors of this kind as you develop your code, rather than causing difficult-to-debug issues afterwards.


If the function/method you’re developing has the same semantics as what’s in Base, then you should probably extend the pre-existing function from Base: this will help with polymorphism.
On the other hand, you definitely don’t want to extend a function from Base (or any other module) if the method you’re adding does not have the same semantics: that could cause issues later if you try to use a generic function that assumes the semantics to be defined by Base, and can’t know about your specific method. In such cases, you’re much better off defining new functions (probably with a different name, definitely with an appropriate docstring).

2 Likes
8

I’m not exactly sure what you mean by method semantics, it’d be great if you can explain that a bit because googling did not help.
I’m definitely wary of extending functions for duck-typing now. I might keep doing it for unusual composite types I make up, but I definitely don’t want to extend operators for data types that are likely used in other packages (like Dual) and have to fix my code when I want to use those packages too.

9

I think the point it is won’t matter…well until you try to USE those functions, then Julia will let you know that you are overriding/hiding a function. Functions are only brought into “your” namespace when they are used by you.

10

What I call the “semantics” of a function is the sort of “contract” that defines (in natural language) what a function does. Take one, for example; its docstring says:

one(x)

Return a multiplicative identity for x : a value such that one(x)*x == x*one(x) == x.

That is the semantics that every method extending one should implement for specific types. It is very much related to generic code and polymorphism, because this “contract” is what you rely on when you write a (very naive, but generic) power function in this way:

function mypow(x, n)
    result = one(x)
    for i in 1:n
        result *= x
    end
    result
end

This implementation will work for any type which defines a product * and a one method with the correct semantics:

julia> mypow(2, 3) == 2^3
true

julia> mypow(2.1, 3) == 2.1^3
true

       # Because * denotes concatenation for strings
julia> mypow("abc", 3) == "abc"^3 == "abcabcabc"
true
1 Like