Method definition overwritten warning for functions with default arguments

General Usage
1

When defining multiple methods of the same function using default arguments, Julia generates extra methods to handle the syntactic sugar. But it can so happen that Julia will define the same methods for each of the methods I’m defining, effectively overwriting itself.

For example:

julia> bar(a::String = "") = "moo"
bar (generic function with 2 methods)

julia> bar(a::String = "", s::Symbol = :s) = "boo"
WARNING: Method definition bar() in module Main at REPL[12]:1 overwritten at REPL[13]:1.
WARNING: Method definition bar(String) in module Main at REPL[12]:1 overwritten at REPL[13]:1.
bar (generic function with 3 methods)

julia> bar(i::Int = 2) = "zee"
WARNING: Method definition bar() in module Main at REPL[13]:1 overwritten at REPL[15]:1.
bar (generic function with 4 methods)

The warning should be hidden since it’s part of Julia’s internals - it won’t help the developer and on the contrary, it’s confusing. I stumbled onto this in a scenario where I was myself dynamically generating various methods and it sent me on a wild goose chase.

2 Likes
2

It’s not internal in the sense that it has observable effect that you’ll never be called the bar() version of the previous two methods anymore.

3

This warning should definitely not be hidden. The method for bar() is overwritten because you are providing default arguments.

julia> bar(a::String = "") = "moo"
bar (generic function with 2 methods)

julia> bar()
"moo"

julia> bar(a::String = "", s::Symbol = :s) = "boo"
WARNING: Method definition bar() in module Main at REPL[1]:1 overwritten at REPL[3]:1.
WARNING: Method definition bar(String) in module Main at REPL[1]:1 overwritten at REPL[3]:1.
bar (generic function with 3 methods)

julia> bar()
"boo"
1 Like
4

Also note that an alternative way is to treat method with default argument as a single method instead of multiple ones like what we do now. In that case, bar() won’t be overwritten but will be an ambiguity that needs to be resolved. The difference between the two approaches is also observable and not an internal detail and both should be equally valid.

5

Thanks

OK, I see - I agree with your points.

But maybe the behavior can be improved? It now requires lower level understanding of how Julia handles methods with default arguments. And it seems to go against the principle of least surprise. We can conclude that the current implementation has, at least, a usability problem - which goes against the idea of having these methods in the first place (to make the language more friendly, DRY-er and usable for the developer).

@joshday Indeed, it is a serious issue and the warning can’t be hidden as the behavior causes very important and potentially critical side effects (another method might be called).

@yuyichao Maybe that could be a compiler option? Overwrite vs explicitly handle ambiguity? But this would still present the usability problem - it would just replace one problem with another. It will however be safer, which is a big deal.

Just brainstorming here, but would it be possible to have typed methods with 0 arguments?

Let’s take these two simple examples:

julia> baz(s::String = "hello") = s
baz (generic function with 2 methods)

julia> baz(i::Int = 2) = i
WARNING: Method definition baz() in module Main at REPL[19]:1 overwritten at REPL[20]:1.
baz (generic function with 3 methods)

Instead of having baz() with the same signature in both cases, could we have a baz(::String) and a baz(::Int) ?

This way
1/ we’d have distinct methods
2/ we’d have the option of invoking either with the default arguments, by literally calling baz(::String) or baz(::Int)

I guess this would still leave open the issue of baz(::Any) – but it would be more flexible as it would allow defining and invoking multiple methods for baz().

6

Default arguments are always a bit tricky, it reminds me of the infamous python function f(a = []) situation. Julia works quite nicely in comparison, but I see what you’re saying. I think people will be alright as long as they remember that

  • for any method with default arguments, Julia creates a copy which takes less arguments
  • methods are overwritten if they have the same signature

Both of those are useful, easily described, and easily discovered, so I’m happy with it.

It’s an interesting idea with the typed 0-argument case. (Although philosophically I find it hard to think about the type of something that is not there.) You could prepare a package, that would get people interested. You’d probably have to use macros to hook into the existing dispatch functionality, though, so it wouldn’t be entirely simple.

1 Like
7

It’s not lower level since you have to define a behavior for “conflicting” missing argument case. This (optional argument being equivalent to defining multiple methods) is also documented right when it is introduced (http://julia.readthedocs.io/en/latest/manual/functions/#optional-arguments and http://julia.readthedocs.io/en/latest/manual/methods/#man-note-on-optional-and-keyword-arguments) and is also observable if you run methods on any function you’ve defined this way.

What’s the usability problem?

That will only make it worse.

At least that’s not how method overloading work so it would be a big semantic/syntax change. It should at least not be added as a special case for default argument.

1 Like
8

one does not even need to know the internals. the provided set of functions is ambiguous no matter what the internals are. in order to be well defined, for any combination of actual parameters it should be clear which version is to be called. and this is simply not the case if two methods accept zero parameters. this indicates that probably we are actually dealing with two functions here that happens to have the same name. maybe some refactoring is needed.

9

It’s possible this behavior will change. See this issue:
https://github.com/JuliaLang/julia/issues/7357

10

AFAICT that issue is about what to do after finding the right method and is irrelevant to this discussion which is only about how to find the right method.

11

Right. It’s just related in that it will help simplifying this system. Though I agree that in the examples provided by the OP the methods clearly overlap and it doesn’t make sense defining conflicting optional arguments.

12

Thank you all for the feedback, much appreciated.

@yuyichao The usability issue is in the fact that in other languages, defining methods with optional arguments Just Works ™. I’ve been programming in a few other for a while now and never ran into this kind of issue before.

Anyway, now that I fully understand the issue, I can handle it. It makes perfect sense in Julia but I have the tendency to ignore warnings early in the development phase. I think many devs are the same, and this one can bite badly.

I don’t have a strong opinion that something should be done about this - though I would like the typed 0 arguments methods, they’d make a nice addition and seem to fit in with Julia’s typing narrative. Maybe it will make more sense in the future, as the language evolves.

1 Like
13

I guess my surprise comes from the fact that I come from a lifetime of programming in weakly typed languages. These are all about the number of arguments - so methods with 2, 3, 4, etc optional arguments will not “leak” 0 arguments methods that overwrite each other. Arity is a core concept there.

In the same line of thought, this is a fairly low level implementation detail, as I have no idea how ruby, or PHP or JavaScript handle methods with optional arguments. I doubt many devs do, as with more mature / older languages people work with higher level frameworks and DSLs which require a massive time investment and level of mastery of their own.

As Julia will go mainstream it will be more and more about higher level frameworks and DSLs and books about these, and less about Julia itself. Ask yourself how many Rails users (not contributors) have read the ruby manual?

It’s in this context where possible traps need to be avoided - and this is probably an ingredient for large scale adoption.

14

This argument (Julia did something I didn’t expect; this will be a problem when it goes mainstream, because my expectations describe the general population, so you should change the language) comes up here very often, many times each week.

Seemingly, there is a tradeoff between minimizing surprise for users coming with various prior histories of programming languages, and coming up with a small, easy to understand, consistent set of rules (which of course one has to learn, there is no way around that). But when these things are discussed, it turns out that people have a wide variety of intuitions about how things should work (because they worked that way in R/Matlab/Python/Ruby/…), so there is usually no DWIM solution that dominates the simple, consistent set of rules solution. See eg this thread and comment:

4 Likes
15

@Tamas_Papp I agree, at this level, the discussion is too generic. But then again, this is relatively easy to address: what are the main approaches to dealing with methods with optional arguments? Are there any other schools of thought? Is Julia’s approach equally common?

Doing a bit more soul searching during breakfast, I’m actually starting to think that this approach might not be entirely consistent with Julia itself - and that arity should, maybe, be an important part in Julia’s implementation too.

The reason is that in Julia, a function in invoked by passing it a Tuple of arguments. A Tuple, by definition, has its type provided by the number and types of its elements.

When I define a method, I define the type of Tuple it should be invoked with. So in this line of thinking, the method’s signature is defined in relation to the type of the Tuple of arguments it takes. Basically we have a method that is defined by its name the fact that it takes a number of exactly x arguments of exactly the types A, B, C, … .

In this reasoning, Julia defining other methods of different arity and types of its arguments Tuple, means constructing completely different methods altogether, with completely different signatures.

So why not always have a fixed method signature (in terms of the type of its Tuple of arguments – so arity and types) and instead have the Tuples that can be constructed with optional arguments?

I hope I managed to explain properly: it’s basically about having the type of the Tuple of arguments a strict part of a method’s signature and not generating methods that take different types of Tuples. Instead the Tuple itself would be more flexible, maintaining its type but accepting default arguments when constructed.

This could be a dedicated subtype of Tuple, like for example MethodArgumentTuple. And when defining methods with optional arguments, instead of generating methods for all the combinations of arguments, Julia would create a new instance of MethodArgumentTuple that would always have the same type (x elements of type A, B, C, …).

So no more baz but always baz{MethodArgumentTuple{A, B, C}} and the Tuple itself would provide the default arguments.

I guess it’s a variation of baz(::String) with a different approach: it’s not the function but the Tuple of arguments providing the values of the optional arguments.

17

You have to and only have to deal with this issue when you have both multiple dispatch/overloading and optional argument. It’s the combination of the two that forces you to make a choice that’s user visible. Scripting/weakly typed languages usually don’t have the first so it’s not an issue for them. C++ kind of has both and they raise an error as long as your call have more than one matches (more or less) and I don’t think that’s better.

Disclaimer: I don’t know that many languages and it’d be interesting to know how other languages combines the too.

1 Like
18

I am sorry but I don’t think I understand what you are proposing. Each method already has a signature, that is matched for dispatch. It is just that optional arguments define extra methods, again with their own signatures.

BTW, did you read the discussion for the issue @nalimilan linked above? I think it is a reasonable solution.

19

The problem you describe in the original post really cannot solved due to sheer logic: you defined both bar(a::String = "") and bar(a::String = "", s::Symbol = :s). How could the language figure which method bar() should call? Clearly it does not make sense to make all arguments optional for both methods, you have to choose one. That really has nothing to do with Julia’s implementation.

2 Likes
20

@nalimilan By telling Julia what method I’m referring to. It boils back to having missing typed arguments - or making arity and the corresponding types (let’s call this arity+type) an integral part of a method’s signature.

With typed missing arguments, bar() would be bar(::Any). Which would be different from bar(::String) and bar(::String, ::Symbol). It could be invoked as bar(::String) or as bar("Hi!", ::Symbol).

With arity+type notation it could be bar<String,Symbol>() corresponding to bar(a::String = "", s::Symbol = :s). With invocations looking like bar<String,Symbol>("Hi!").

I prefer the esthetics of the 2nd form, which would resemble the current bar{T,N} as bar<T,N> and would allow bar<T,N>{T,N}.

@Tamas_Papp Sorry, my example was too convoluted. I’m basically suggesting what’s above: that the original method signature (arity+types) stays “sticky”. And can be invoked explicitly; versus the current behavior where Julia takes a higher-arity-more-specific method and generates lower-arity-less-specific methods, losing track of what was the original method and creating overwriting methods. Makes sense?

Indeed, I’ve read the issue referenced by @nalimilan - I assume you’re referring to the _hidden_f_ approach. Not sure how this would work with the examples we discussed, when higher-arity-more-specific methods generate less-specific-lower-arity methods?

It seems to me that

bar(s::String = "moo")

and

bar(i::Int = 2)

would generate

bar() = _hidden_bar_("moo")

and

bar() = _hidden_bar_(2)

which would still be overwriting implementations.

Unless we go back again to my suggestion:

bar(::String) = _hidden_bar_("moo")

or

bar<String>() = _hidden_bar_("moo")

But of course, if we have bar(::String) or bar<String>() we don’t need _hidden_bar_ anymore.

@yuyichao I understand. My point is no longer about the initial question (thank you all for clearing that out, btw), but rather about “can we think of a better way of doing this?”. In the end, the decision belongs to the core team - I just hope to offer a different (fresh?) perspective.

It took a bit of a detour but I’d say that the above defines my view of a possible implementation. Tagging method definitions with arity+type information which would allow the simultaneous definition of multiple methods without overwrites and would also allow disambiguation and explicit invocation. A possible syntax would be:

f<T,N>(x::T = T(), y::N = N()) = ...

So for

bar(s = "", i = 42) = # ...

Julia would generate the tagged methods

bar<String,Int64>(s::String, i::Int64) 
bar<String,Int64>(s::String) 
bar<String,Int64>()

And for

bar(s = "", sy = :s) = :x

Julia would generate

bar<String,Symbol>(s::String, sy::Symbol) 
bar<String,Symbol>(s::String) 
bar<String,Symbol>()

For disambiguation, where necessary, we could invoke the desired method as

bar<String,Int>()

The tagged invocation would be optional and only required for disambiguation, plain bar() would still work if no ambiguities.

How other languages deal with this is indeed quite intriguing - I’ll do a bit of research into that, I’m curious too. If anything useful comes out, I’ll report back.

21

OK, I didn’t understand your proposal then. But I don’t see how this would be an improvement over the current semantics. When there is a collision, it replaces accidental overwriting of methods with accidentally calling the other method, unless the ambiguity is resolved with the <> syntax, which would then introduce a new syntax for the sole purpose of solving a problem that replaced another problem.

1 Like