Unexpected sensitivity to order of method definitions

General Usage
1

The following works like a charm:

function f(a; b)
    a + b
end

function f(a)
    a + a
end

julia> f(2)
4
julia> f(2,b=1)
3

But reversing the order results in an error:

function f(a)
    a + a
end

function f(a; b)
    a + b
end

julia> f(2)
ERROR: UndefKeywordError: keyword argument `b` not assigned

This seems like bug but maybe I’m missing something? I’m running Julia 1.9.1.

1 Like
2

The definitions override each other, I think. methods(f) shows the list of methods that have been defined.

1 Like
3

Keyword arguments don’t participate in dispatch

3 Likes
4

There’s a hint in the method count displayed when you define the functions:

julia> f(a) = a + a
f (generic function with 1 method)

julia> f(a; b) = a + b
f (generic function with 1 method)

As you see there’s only 1 method for f after the second definition, which tells you that the first definition was overwritten.

5

Ok but somehow the first version still works, the first definition doesn’t actually get overwritten even though methods(f) says there’s only 1 method…?

2 Likes
6

Not sure what you mean.

Anyway, you can start Julia with --warn-overwrite=yes to make it warn on method overwrite. I always start it like that.

1 Like
7

I think Tuuka’s point is that:

C:\>julia -q --warn-overwrite=yes

julia> f(a; b) = a + b
f (generic function with 1 method)

julia> f(a) = a + a
WARNING: Method definition f(Any) in module Main at REPL[1]:1 overwritten at REPL[2]:1.
f (generic function with 1 method)

julia> f(2, b = 1)
3

this shouldn’t work if f(a, b = 1) really got overwritten. I think this is:

which was filed over two years ago but didn’t get any responses.

7 Likes
8

Indeed. It doesn’t get overwritten, they are both there… Confusing.

julia> f(a; b) = a + b
f (generic function with 1 method)

julia> f(a) = a + a
f (generic function with 1 method)

julia> f(2, b = 1)
3

julia> methods(f)
# 1 method for generic function "f" from Main:
 [1] f(a; b)
     @ REPL[2]:1

julia> f(a) = 10a
f (generic function with 1 method)

julia> f(20)
200

julia> f(20, b=1)
21
9

This example is in fact almost exactly demonstrated in this comment on issue 9498, only the keyword argument has a default value. (The issue’s first example differs because the positional annotations can be different, so the methods shouldn’t override each other. methods(f) does show them as separate methods, and it’s more intuitive that a keyworded call can’t use a method with no keywords even if the positional annotation is more specific).

Redoing the demo with methods(f), keyword method first 
julia> function f(a; b)  a + b  end;

julia> methods(f)
# 1 method for generic function "f":
[1] f(a; b) in Main at REPL[1]:1

julia> function f(a)  a + a  end;

julia> methods(f)
# 1 method for generic function "f":
[1] f(a; b) in Main at REPL[3]:1

julia> f(2), f(2; b=1)
(4, 3)
Demo but keyword method second in a different session 
julia> function f(a)  a + a  end;

julia> methods(f)
# 1 method for generic function "f":
[1] f(a) in Main at REPL[1]:1

julia> function f(a; b)  a + b  end;

julia> methods(f)
# 1 method for generic function "f":
[1] f(a; b) in Main at REPL[3]:1

julia> f(2; b=1)
3

julia> f(2)
ERROR: UndefKeywordError: keyword argument b not assigned

If you’ve noticed, it certainly seems as if the keyworded method actually had 2 methods, 1 throwing the error with no keywords. This would make a little more sense; in the first case, the error-throwing method was overwritten with a proper implementation, and vice versa in the second case. But where is it?

Let’s try just the keyworded method with a default value this time so we don’t throw errors. I will use ... to omit some printed lines for brevity.

julia> function f(a; b=0) a+b end;

julia> methods(f)
# 1 method for generic function "f":
[1] f(a; b) in Main at REPL[14]:1

julia> @code_warntype f(1)
MethodInstance for f(::Int64)
  from f(a; b) in Main at REPL[14]:1
...
1 ─ %1 = Main.:(var"#f#3")(0, #self#, a)::Int64
└──      return %1

julia> @code_warntype f(1; b=10)
MethodInstance for (::var"#f##kw")(::NamedTuple{(:b,), Tuple{Int64}}, ::typeof(f), ::Int64)
  from (::var"#f##kw")(::Any, ::typeof(f), a) in Main at REPL[14]:1
...
5 ┄ %16 = Main.:(var"#f#3")(b, @_3, a)::Int64
└──       return %16

The keywordless call directly lists the 1 f(a; b) method, but the keyworded call does not. It lists a var"#f##kw" functor type. Both calls have the callee var"#f#3, only the keywordless call has the default value.

julia> methods(var"#f#3")
# 1 method for generic function "#f#3":
[1] var"#f#3"(b, ::typeof(f), a) in Main at REPL[14]:1

julia> var"#f#3"(10, f, 1)
11

julia> methods(var"#f##kw") # huh, constructor was omitted
# 0 methods for type constructor

julia> Base.issingletontype(var"#f##kw") # singleton.instance loophole!
true

julia> methods(var"#f##kw".instance)
# 1 method for anonymous function "f##kw":
[1] (::var"#f##kw")(::Any, ::typeof(f), a) in Main at REPL[14]:1

julia> var"#f##kw".instance((b=10,), f, 1)
11

So there you have it, keyworded calls are handled by a hidden singleton functor instance, and the keywordless calls are directly handled by the declared keyword method. The latter clashes with position-only methods, independently of the former. Both forward to a callee, which is what you want to @code_warntype to actually probe type instabilities (or use Cthulhu.jl to descend through callees).

1 Like
10

Thanks for all the comments! Seems like this is a known issue and the problem is that handling keyword arguments is a little bit messy in Julia right now, making it harder to fix it. I’ll try not to use keyword arguments to affect dispatch (even though sometimes it would be quite convenient if it worked).

11

It’s fine, just don’t overwrite any methods. Overwriting methods is basically only useful while hacking anyway.

1 Like
12

It can be intimidating, but being aware of the limitations of mixing keywords and multimethod features helps a lot. One more thing I can mention is stick to the best practice of keeping keywords consistent across methods; ambiguities happen for the ones with the same number of positional arguments and related types, see the example below.

julia> bar(::Float64) = 0
bar (generic function with 1 method)

julia> bar(::Int; a) = 1        # if ::String, error below would not happen
bar (generic function with 2 methods)

julia> bar(::Number; a, b) = 2  # Int<:Number causes error below
bar (generic function with 3 methods)

julia> bar(1.0; a=0, b=0) # cannot use `bar(::Float64)`
2

julia> bar(1; a=0, b=0)
ERROR: MethodError: no method matching bar(::Int64; a=0, b=0)
Closest candidates are:
  bar(::Int64; a) at REPL[2]:1 got unsupported keyword argument "b"
  bar(::Number; a, b) at REPL[3]:1
  bar(::Float64) at REPL[1]:1 got unsupported keyword arguments "a", "b"

Not sure if you also meant that you hope that keyword arguments can eventually distinguish methods by arity, type, and keyword, like how positional ones do by arity, type, and order, but just in case, that really is infeasible. At first it seems possible that keywords can be sorted by name, then it reduces to typical order-based dispatch. However, more methods that vary by keywords makes the above dispatch headaches even worse, and default values would cause combinatorial explosion. Right now, default values in positional arguments make multiple methods that scale linearly N+1 e.g. f(a=1, b=2) makes f(), f(a), f(a, b). If you apply this to keyword arguments that can be reordered, that becomes sum(factorial(N)/factorial(N-i) for i in 0:N) e.g. f(;a=1, b=2) makes f(), f(;a), f(;b), f(;a, b), f(;b, a). The only way to avoid that combinatorial explosion is to force 1 method to handle all those possibilities and disallow other methods from handling any subset of them; silver lining is that the compiler can still specialize that 1 method on the various combinations of keywords you provide calls. It’s not a coincidence that languages with more flexible keyword arguments also lack multimethods or overloaded functions.

2 Likes