If you define a new number-like thing that supports multiplication, extending Base.:* is the correct thing to do. That is so that e.g. code in LinearAlgebra for matrix multiply can work on the new type.
1 Like
What if I want to define a not a number-like thing that supports multiplication? Or a different type of multiplication? Either I add it to Base.:*, I qualify every use of * with a namespace, or I don’t do a using Base to operate with my new type (in which case, none of the other Base type stuff can work, even though it is a completely different sense of multiplication than what I had in mind, for a completely different set of types).
Yes, this needed dependency of two packages which should not depend on each other is exactly the
problem that I perceive. I think that an example I gave should also need to be solved:
Suppose you have defined a function foo in package A which has 10 methods, so that using A
you get a rich functionality of foo. Now, for code design reasons, you decide that you should
split A in two, A and B, to have cleaner code, and that 5 of the 10 methods of foo should go in B.
I really believe that it should be possible to do this so that using both A and B is completely equivalent to using A before the split.
But what about a scenario where we don’t have the using statements anymore? As witnessed by @StefanKarpinski’s comment (which I agree with for the most part), there is a sense in which using is bad practice, at least for production quality code. I made a similar argument in this comment. In a universe where using ThisOrThatPackage statements have been replaced with precise, explicit import statements, I think argument-dependent lookup makes a lot of sense.
Here’s a slightly modified version of my example from earlier. What if (with ADL) the following was a thing:
module LinearAlgebra
struct Vec2{T}
x::T
y::T
end
*(x::Number, v::Vec2) = Vec2(x * v.x, y * v.y)
end
module FixedPointNumbers
struct FixedPointNumber end # just as an example
*(x::FixedPointNumber, y::Float64) = # implement me
end
module UserModule
# imports, not using statements!
# (or equivalently, get rid of export statements in all packages)
import FixedPointNumbers: FixedPointNumber
import LinearAlgebra: Vec2
v = Vec2(1.0, 2.0)
x = FixedPointNumber()
x * v
end
The call x * v would resolve to LinearAlgebra.*, because Vec2 is defined in LinearAlgebra, and there is a function LinearAlgebra.* with a method that specifically takes a Vec2 as one of its arguments, by ADL rules. Similarly, the call to * in LinearAlgebra.* would find FixedPointNumbers.* because x is a FixedPointNumber, and there is a function FixedPointNumbers.* that specifically takes a FixedPointNumber.
1 Like
By what rule can LinearAlgebra see the FixedPointNumber definition of *, unless that definition is somehow part of a function that LinearAlgebra imports? One way would be to have a single namespace of function names. But then if two packages define f(x::Any), calls to those methods always need to be qualified. One line of argument says that situation would be rare, but I’m not so sure.
Yes! That is the goal. No guessing of which interface you meant. And if there is an existing f(x::Any) existed in any library I am using and I create my own, all sorts of ambiguity errors would arise.
No, you are missing some options. import MyPkg: * will let you use it unqualified even if it is not added to Base.:*. We could make that even easier by allowing using X to shadow the implicit using Base.
I think this nails it.
Which, by the way, is exactly the style in C++. You are really never supposed to use using for everything in a namespace (not even in short sample code), and using std; is verboten in production code.
Ok, this seems to me to refer to a design where there is a single namespace of function names. Now, I don’t think such a design is completely crazy. We could say that every exported function with the same name is the same function. (This is more or less the suggestion in Common pool for methods as a way to solve common names in different packages · Issue #2327 · JuliaLang/julia · GitHub).
If we did this naively today, you couldn’t have more than one f(x::Any) — one definition would simply overwrite the other. So we would need some new mechanism that keeps both, and makes Foo.:* some object that means “call *, but only seeing methods defined in Foo”. That would be tricky, since the code for Foo might just be using the universal, unqualified *, so Foo.:* would not mean “the value of * inside the Foo module”, which is what it always means now. But it’s probably possible to implement somehow.
3 Likes
That’s not how it works in C++. In C++, each function would still live in its own namespace, and if you want to be explicit, you can call somepackage::foo without any C++ using statements, and without foo living in a global namespace. I think section 1.1 of this document by Herb Sutter explains it very well: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2103.pdf.
I’m amused by this thread as this is something I wanted once (https://github.com/JuliaLang/julia/issues/2327), but as I’ve used Julia the the Julia package ecosystem, I’ve come to think that it’s a bad idea. Using a common pool or single namespace is too limiting, and it discourages use of untyped arguments like f(x::Any).
I like Jeff’s idea that methods that are specifically imported override the Base function of the same name.
4 Likes
But if every call f(x) looks into the namespace of x’s type to find f, then every call to f can see every definition of f, so these namespaces are not separate in a terribly strong sense.
Sutter would say (his ‘interface principle’): all functions f that both ‘mention’ (in a certain, precisely defined way) typeof(x) (say X) and are defined in the same module as X (say XModule) are logically a part of X. So f is only visible without the explicit imports by virtue of the user’s use of a value of type X. It’s clear that, without using statements, the user must have intended XModule.f, because XModule.f is part of the interface of X, and should thus come along for the ride. And again, if you want to be specific, you can just call XModule.f specifically.
For languages without nonmember functions, this is a no-brainer (imagine in Java if you had to import not only a class, but also all of the methods of that class that you were going to use). C++ just generalizes the concept. Julia has only nonmember functions, which makes it seem very different, but maybe it shouldn’t be.
2 Likes
I think the convention is that the language is in English, therefore functions with the same name
mean usually the same concept therefore it is reasonable to merge them if possible.
I don’t know if this is possible but if there could be a way to determine ,for two exported functions with the same name, whether they are mutually exclusive in their type signatures , then I think they could be merged automatically and if not issue a warning and have the programmer qualify both.
Just to be clear, though, an ADL based implementation is the polar opposite of a single namespace. It even makes it practical to write code without any using Base. C++ is perfectly able to handle untyped generic functions with ADL (e.g. use the auto keyword and completely unconstrained template parameters). Of course, you can’t use 2 untyped functions with the same name at the same time without qualification, but that is a good thing.
Ah, this is very useful. This touches on the heart of the problem. Julia method applicability is based on subtyping, and not on something like whether a definition “mentions” a certain type. In fact, we even want to know the set of applicable methods without knowing anything about the type of x — what does methods(f) or methods(f, (Any,)) print?
Just to make sure I understand: if C++ had an Any type, and XModule contained f(x::Any), the call f(x) somewhere else (in a module that doesn’t import XModule) could not call that method? If that’s true, I’d say it’s fundamentally incompatible with how julia works. One way to see it is as a kind of non-monotonicity: you can increase the set of applicable methods by narrowing an argument type.
2 Likes
Exactly.
Is that really different from C++'s templated functions and template specializations though? C++ makes ADL work for these. In fact, Sutter’s ‘modest proposal’ PDF from my earlier comment addressed exactly the issue that C++ used to be too liberal in applying ADL in the face of templated functions.
Personally, I am way past thinking that a naive tweak to the existing setup is possible. As this is a fundamental rethinking of Julia’s scoping rules to help it scale to the next level and enable coding without using Base, I think it is better to do a thorough design to see if it is possible to design for the next major version. Until then, I think some workarounds to make the scoping less surprising for new users is worth examining.
As @tkoolen says, this is a no-brainer to design and implement in languages with only member functions (where many users might come from), but was tricky for C++ and others. All of the papers and discussion on ADL in C++ are extremely valuable, since they had to deal with similar issues.
Also, even if ADL makes sense, I think that @StefanKarpinski made an excellent point that it is trickier to implement in a dynamic language, which makes getting it right even more important. Luckily, with ADL you should always err on the side of strictness and forcing manual ambiguity resolution, but even detecting ambiguity gets tough with functions being added/removed from namespaces.
Agreed. That and how to handle functions as first-class citizens.
Ok, I finally understand this. It took me a while, because this C++ approach is so completely at odds with the lisp worldview. The lisp view of f(x, y) is that first f, x, and y are evaluated. In fact they can each be arbitrary expressions and don’t have to be names/symbols. Then the object resulting from evaluating f is used to do a call. That object always does the same thing no matter how you got it. E.g. f(x, y) can be rewritten to
stuff = [f]
stuff[1](x, y)
Now, it’s possible that f could be an object that implements a form of ADL over its arguments. But it’s these objects that would populate the “single global namespace” of function names — there has to be some namespace that maps the name f to an object that does what you want, and you have to be able to see that binding.
8 Likes