I also agree that upstream multiple dispatch ambiguities can result in errors that are horrible for end-users, and I’d say that it’s not nearly as easy to resolve these problems in the real world as it is with toy examples. Sure, all code involved can technically be “discovered” programmatically, and is actually “there”, but come on - it’s ridiculous to expect a naive end-user to be able to debug a
MethodError across multiple packages where the package authors themselves didn’t even foresee the problem.
For example, let’s say our user wants to solve some optimization problem, where PkgA provides some kernel they’d like to use to define the objective, and PkgB provides the solver:
f(x, y) = PkgA.g(x, y) * y
PkgB.optimize(f, x, y; use_ad = true)
Code like the above can easily throw ambiguity-related errors when
PkgA.g employs an upstream package that happens to be ambiguous with the AD package used by
PkgB. This gets even worse when - as @tkoolen pointed out - such code is actually in a package that then other people/packages depend on.
This specific conversation has been mostly about
Number types, for which there is a
promotion mechanism and it is often possible to assume efficient convertibility between subtypes. However, this is much more of a nasty problem to deal with when you don’t even have a
promotion mechanism to fall back on, e.g. you’re not working with types that are not efficiently convertible to one another. In this case, most hacks around it end up being nasty and load-order dependent, as was mentioned previously.
The good news is that I think we actually have decent ideas for solutions to this problem at this point. As mentioned earlier, Cassette can solve this problem by enabling package authors to specify the “context” in which their code is running, and have that “context” always take precedence.
For example (
B are provided by different package authors that don’t know about each other):
can be explicitly resolved in order of context application, e.g.
PkgB.doBthing((x, y) -> PkgA.doAthing(f, A(x), y), x, B(y))
would presumably result in (using psuedocode for notation):
This works even if the caller is calling these packages indirectly, like in our optimization example above.
Of course, whether or not Cassette actually makes sense for either
B is dependent on what you were trying to achieve with your
A() or your
B() in the first place; but generally, only one of them needs to be “cassette-ified” in order for this particular case (binary ambiguities) to be considered resolved.
There’s also a less general, but non-Cassette solution in the form of https://github.com/peterahrens/Hyperspecialize.jl.