Method definition overwritten warning for functions with default arguments

question

#9

It’s possible this behavior will change. See this issue:


#10

AFAICT that issue is about what to do after finding the right method and is irrelevant to this discussion which is only about how to find the right method.


#11

Right. It’s just related in that it will help simplifying this system. Though I agree that in the examples provided by the OP the methods clearly overlap and it doesn’t make sense defining conflicting optional arguments.


#12

Thank you all for the feedback, much appreciated.

@yuyichao The usability issue is in the fact that in other languages, defining methods with optional arguments Just Works ™. I’ve been programming in a few other for a while now and never ran into this kind of issue before.

Anyway, now that I fully understand the issue, I can handle it. It makes perfect sense in Julia but I have the tendency to ignore warnings early in the development phase. I think many devs are the same, and this one can bite badly.

I don’t have a strong opinion that something should be done about this - though I would like the typed 0 arguments methods, they’d make a nice addition and seem to fit in with Julia’s typing narrative. Maybe it will make more sense in the future, as the language evolves.


#13

I guess my surprise comes from the fact that I come from a lifetime of programming in weakly typed languages. These are all about the number of arguments - so methods with 2, 3, 4, etc optional arguments will not “leak” 0 arguments methods that overwrite each other. Arity is a core concept there.

In the same line of thought, this is a fairly low level implementation detail, as I have no idea how ruby, or PHP or JavaScript handle methods with optional arguments. I doubt many devs do, as with more mature / older languages people work with higher level frameworks and DSLs which require a massive time investment and level of mastery of their own.

As Julia will go mainstream it will be more and more about higher level frameworks and DSLs and books about these, and less about Julia itself. Ask yourself how many Rails users (not contributors) have read the ruby manual?

It’s in this context where possible traps need to be avoided - and this is probably an ingredient for large scale adoption.


#14

This argument (Julia did something I didn’t expect; this will be a problem when it goes mainstream, because my expectations describe the general population, so you should change the language) comes up here very often, many times each week.

Seemingly, there is a tradeoff between minimizing surprise for users coming with various prior histories of programming languages, and coming up with a small, easy to understand, consistent set of rules (which of course one has to learn, there is no way around that). But when these things are discussed, it turns out that people have a wide variety of intuitions about how things should work (because they worked that way in R/Matlab/Python/Ruby/…), so there is usually no DWIM solution that dominates the simple, consistent set of rules solution. See eg this thread and comment:


#15

@Tamas_Papp I agree, at this level, the discussion is too generic. But then again, this is relatively easy to address: what are the main approaches to dealing with methods with optional arguments? Are there any other schools of thought? Is Julia’s approach equally common?

Doing a bit more soul searching during breakfast, I’m actually starting to think that this approach might not be entirely consistent with Julia itself - and that arity should, maybe, be an important part in Julia’s implementation too.

The reason is that in Julia, a function in invoked by passing it a Tuple of arguments. A Tuple, by definition, has its type provided by the number and types of its elements.

When I define a method, I define the type of Tuple it should be invoked with. So in this line of thinking, the method’s signature is defined in relation to the type of the Tuple of arguments it takes. Basically we have a method that is defined by its name the fact that it takes a number of exactly x arguments of exactly the types A, B, C, … .

In this reasoning, Julia defining other methods of different arity and types of its arguments Tuple, means constructing completely different methods altogether, with completely different signatures.

So why not always have a fixed method signature (in terms of the type of its Tuple of arguments – so arity and types) and instead have the Tuples that can be constructed with optional arguments?

I hope I managed to explain properly: it’s basically about having the type of the Tuple of arguments a strict part of a method’s signature and not generating methods that take different types of Tuples. Instead the Tuple itself would be more flexible, maintaining its type but accepting default arguments when constructed.

This could be a dedicated subtype of Tuple, like for example MethodArgumentTuple. And when defining methods with optional arguments, instead of generating methods for all the combinations of arguments, Julia would create a new instance of MethodArgumentTuple that would always have the same type (x elements of type A, B, C, …).

So no more baz but always baz{MethodArgumentTuple{A, B, C}} and the Tuple itself would provide the default arguments.

I guess it’s a variation of baz(::String) with a different approach: it’s not the function but the Tuple of arguments providing the values of the optional arguments.


#17

You have to and only have to deal with this issue when you have both multiple dispatch/overloading and optional argument. It’s the combination of the two that forces you to make a choice that’s user visible. Scripting/weakly typed languages usually don’t have the first so it’s not an issue for them. C++ kind of has both and they raise an error as long as your call have more than one matches (more or less) and I don’t think that’s better.

Disclaimer: I don’t know that many languages and it’d be interesting to know how other languages combines the too.


#18

I am sorry but I don’t think I understand what you are proposing. Each method already has a signature, that is matched for dispatch. It is just that optional arguments define extra methods, again with their own signatures.

BTW, did you read the discussion for the issue @nalimilan linked above? I think it is a reasonable solution.


#19

The problem you describe in the original post really cannot solved due to sheer logic: you defined both bar(a::String = "") and bar(a::String = "", s::Symbol = :s). How could the language figure which method bar() should call? Clearly it does not make sense to make all arguments optional for both methods, you have to choose one. That really has nothing to do with Julia’s implementation.


#20

@nalimilan By telling Julia what method I’m referring to. It boils back to having missing typed arguments - or making arity and the corresponding types (let’s call this arity+type) an integral part of a method’s signature.

With typed missing arguments, bar() would be bar(::Any). Which would be different from bar(::String) and bar(::String, ::Symbol). It could be invoked as bar(::String) or as bar("Hi!", ::Symbol).

With arity+type notation it could be bar<String,Symbol>() corresponding to bar(a::String = "", s::Symbol = :s). With invocations looking like bar<String,Symbol>("Hi!").

I prefer the esthetics of the 2nd form, which would resemble the current bar{T,N} as bar<T,N> and would allow bar<T,N>{T,N}.

@Tamas_Papp Sorry, my example was too convoluted. I’m basically suggesting what’s above: that the original method signature (arity+types) stays “sticky”. And can be invoked explicitly; versus the current behavior where Julia takes a higher-arity-more-specific method and generates lower-arity-less-specific methods, losing track of what was the original method and creating overwriting methods. Makes sense?

Indeed, I’ve read the issue referenced by @nalimilan - I assume you’re referring to the _hidden_f_ approach. Not sure how this would work with the examples we discussed, when higher-arity-more-specific methods generate less-specific-lower-arity methods?

It seems to me that

bar(s::String = "moo") 

and

bar(i::Int = 2)

would generate

bar() = _hidden_bar_("moo")

and

bar() = _hidden_bar_(2)

which would still be overwriting implementations.

Unless we go back again to my suggestion:

bar(::String) = _hidden_bar_("moo")

or

bar<String>() = _hidden_bar_("moo")

But of course, if we have bar(::String) or bar<String>() we don’t need _hidden_bar_ anymore.

@yuyichao I understand. My point is no longer about the initial question (thank you all for clearing that out, btw), but rather about “can we think of a better way of doing this?”. In the end, the decision belongs to the core team - I just hope to offer a different (fresh?) perspective.

It took a bit of a detour but I’d say that the above defines my view of a possible implementation. Tagging method definitions with arity+type information which would allow the simultaneous definition of multiple methods without overwrites and would also allow disambiguation and explicit invocation. A possible syntax would be:

f<T,N>(x::T = T(), y::N = N()) = ... 

So for

bar(s = "", i = 42) = # ...

Julia would generate the tagged methods

bar<String,Int64>(s::String, i::Int64) 
bar<String,Int64>(s::String) 
bar<String,Int64>() 

And for

bar(s = "", sy = :s) = :x

Julia would generate

bar<String,Symbol>(s::String, sy::Symbol) 
bar<String,Symbol>(s::String) 
bar<String,Symbol>() 

For disambiguation, where necessary, we could invoke the desired method as

bar<String,Int>()

The tagged invocation would be optional and only required for disambiguation, plain bar() would still work if no ambiguities.


How other languages deal with this is indeed quite intriguing - I’ll do a bit of research into that, I’m curious too. If anything useful comes out, I’ll report back.


#21

OK, I didn’t understand your proposal then. But I don’t see how this would be an improvement over the current semantics. When there is a collision, it replaces accidental overwriting of methods with accidentally calling the other method, unless the ambiguity is resolved with the <> syntax, which would then introduce a new syntax for the sole purpose of solving a problem that replaced another problem.


#22

i still believe that this is a refactoring situation. if you are willing to decorate your function with the input types, you could just as well add some postfix, like barsym and barindex or whatever the integer means.


#23

@Tamas_Papp The current semantics simply don’t permit the definition of multiple methods, which causes the overwrites in the first place. Then this could possibly be used to solve other related problems, such as the one we talked about earlier (with the _hidden_f).

@pint That would considerably reduce the flexibility of the API, especially when creating DSLs. My issue appeared while writing a DSL for a HTML-like (or rather HAML-like) templating engine. My goal was to come up with an easy to use syntax and provide helpful defaults. I dynamically generate methods corresponding to HTML elements, such as span(), p(), div(), etc. Using the do syntax this ends up looking very much like plain HTML. Doing things like div_empty or div_with_attrs or however you want to call them, would defeat the purpose.


#24

I must admit I don’t really understand your proposal, but why can’t you do this:

bar(a::String) = "moo"
bar(i::Int) = "zee"

bar(::Type{String}) = bar("")
bar(::Type{Int}) = bar(2)

julia> bar(String)
"moo"
julia> bar(Int)
"zee"

?


#25

that does not explain to me why do you want bar() to exist in two versions, one for omitted integer, one for omitted symbol (or any other signatures).

but if so, you can still abandon defaults, and provide your own default constant. like

const DEFBARINT = 7
bar(n::Int) = ...

bar(DEFBARINT)
# instead of bar<Int>() or bar(::Int)

after all, this is what “i want an int here, but i don’t tell what” actually means


#26

Elaborating a bit:

What you want is for the user of your code to be able to specify the type when s/he wishes to use a default value, right? I prefer my own suggestion to yours in that case: that the user calls

bar("Hello", Symbol)

to indicate that the default Symbol value should be used. In that case you are left with needing a convenient way of generating those extra definitions. As far as I understand, that is what macros are for:

You make a macro with catchy name, like @usetypesasdefaults, and then

@usetypesasdefaults foo(a::String = "", b::Int = 2) = [...]

gets rewritten to

foo(a::String, b::Int) = [...]
foo(::Type{String}, b::Int) = foo("", b)
foo(a::String, ::Type{Int}) = foo(a, 2)
foo(::Type{String}, ::Type{Int}) = foo("", 2)

For many inputs this can cause an explosion of methods, so maybe it could re-write to

function foo(a::Union{String, Type{String}, b::Union{Int, Type{Int}})
    if isa(a, DataType)
        a = ""
    end
    if isa(b, DataType)
        b = 2
    end
    [...]
end

This is just off the top of my head. There are probably ways to simplify this.

Edit: One other advantage (aside from not breaking most Julia code ever written) is that the user now only has to specify the type of the missing input, instead of all the types (which might not even be known before run-time!!)


#27

I don’t really want bar() to exist in 2 versions – that was just a generalization (simplification) as the discussion moved up from my initial question.

My exact use case is this: I build a DSL for constructing HTML pages - a templating language similar to HAML. I have some functions that generate an HTML document. And I want them to be used like this:

DIV() do 
  H1() do 
    "Hello world!"
  end
end

But of course, an important part of HTML are the attributes. I want to be able to pass them. So I want to do:

A([:href => "http://...", :class => "menu", :onclick => "loginModal()"]) do 
  "Login"
end

But then I wanted to make it easy on the user of the API and avoid that she forgets the parenthesis so I made them optional, allowing:

H1(:class => "main", :style => "border: 1px solid red") do 
  "Welcome!"
end

Finally, there are empty HTML elements – I need these too: HR(), BR() etc. Since they’re void it means they can’t contain any other HTML element so they won’t need the first argument, the Function; but can very well have the 2nd, for the attributes.

In my code I have something like this:

const NORMAL_ELEMENTS = [ :HTML, :HEAD, :BODY, :TITLE, :STYLE, :ADDRESS, :ARTICLE, :ASIDE, :FOOTER, ... ]

function attributes(attrs::Vector{Pair{Symbol,String}} = Vector{Pair{Symbol,String}}()) :: Vector{String}
  a = String[]
  for (k,v) in attrs
    push!(a, "$(k)=\"$(v)\"")
  end

  a
end

function normal_element(f::Function, elem::String, attrs::Vector{Pair{Symbol,String}} = Vector{Pair{Symbol,String}}()) :: HTMLString
  a = attributes(attrs)

  """\n<$( string(lowercase(elem)) * (! isempty(a) ? (" " * join(a, " ")) : "") )>\n$(f())\n</$( string(lowercase(elem)) )>"""
end

function register_elements()
  for elem_name in NORMAL_ELEMENTS
    f_body = """
      function $elem_name(f::Function = ()->"", attrs::Vector{Pair{Symbol,String}} = Vector{Pair{Symbol,String}}()) :: HTMLString
        \"\"\"\$(normal_element(f, "$(string(elem_name))", attrs))\"\"\"
      end
    """

    f_body |> parse |> eval

    f_body = """
      function $elem_name(f::Function = ()->"", attrs::Pair{Symbol,String}...) :: HTMLString
        \"\"\"\$($elem_name(f, [attrs...]))\"\"\"
      end
    """

    f_body |> parse |> eval
  end
end

It works quite well:

julia> Flax.DIV(:class => "main", :onclick => "go()") do
       "hello"
       end
"<div class=\"main\" onclick=\"go()\">\nhello\n</div>"

julia> Flax.HR()
"<hr>"

julia> Flax.BR(:class => "foo")
"<br class=\"foo\">"

but it causes Julia to output a long list of warnings as it generates overwriting methods. This is how it all started.

Now, I can find a way around them, I guess the simplest being by removing the version with the Vector. But I was wondering if there isn’t a way of avoiding that Julia creates this situation in the first place.


#28

Would standard Julia keyword arguments work?

H1(class="main", style="border")

The Parameters.jl package may also be relevant:


#29

That’s interesting - and it does look pretty. I’ll try it out, thanks