Julep: Taking multiple dispatch,export,import,binary compilation seriously

Internals & Design
81

No doubt it is the closest, and perhaps it is the only possible approach! In Haskell you can’t even have the same field names between types.

I agree. If people don’t use the same operators (or function names) as other packages (or base) and use a largely disjoint set of operators from others, then it works great. The question is how sustainable that it with lots of packages, how teachable it is to people who have fragiliity in their own method name choices when things break as they decide to use a new package, and whether it is basically encouraging people to move towards C style names for everything (i.e. the obvious permutation to avoid future and current namespace clashes with a function is to just put the name of the package into function, making namespaces somewhat pointless).

Maybe this is as good as it can get with a programming language with first-class functions, but I would keep your mind open that there could be a better (and largely backwards compatible) way in the future by looking to what over languages have done.

Anyways, this has been done to death at this point. One of my goals was the hope that people would understand that these namespace issues are not inherent to multiple-dispatch, but a consequence of design decisions in Julia (and, possibly, design decisions where there are no foreseeable alternatives). But it is not a case of “with ADL you can’t do generic programming correctly”. In fact, ADL was created precisely to enable generic programming with namespaces and operators.

1 Like
82

Everything in Julia has its namespace as part of the name. Another way to pick a different name is to just have it in a different namespace. there’s DifferentialEquations.solve and JuMP.solve. They do different things. import if you want to use the full name (or don’t: you can still namespace with using). And you can’t say this cannot be done because… people have been doing it successfully for a long time. You state

But the reality is the following:

and I can keep going. Just look at the Github search results yourself. If you want to shorten it, do:

import DifferentialEquations
const de = DifferentialEquations
de.solve(...)

The packages are already well-namedspaced, which makes them fit perfectly into the Python-style which can be seen in the diffeqpy Python bindings:

But namespacing and function extension are different concepts. If JuMP’s solve and DiffEq’s solve are doing the same thing, then we should merge them to be different dispatches on the same function. Since that’s not always the case (and sometimes dispatches can non-trivially overlap: JuMP could define solve(::Matrix) and DiffEq can define solve(::AbstractMatrix) which don’t overlap but has a DiffEq-assumption on how solve(::Matrix) works which can be different), any kind of function merging can be dangerous or at least non-trivial.

Now I know I didn’t document it like this. I didn’t document it putting de. in front (in the Julia docs) but in isolated code you don’t have to. If you think this is a teaching issue, then this is a documentation issue / opinion but again not a language issue.

This brings us to something else:

Yes, Base is special. That’s what makes Julia nice to work with. We all have the same basic mathematical functions which we can use. We can expect everyone knows them. This fact makes Julia much easier to teach than something like Python/R where you have to search for libraries to do even basic things like linear algebra! The other language equivalent here is to say that you want to overwrite / monkey patch some basic NumPy operation. That’s essentially “Julia’s Base” in a language that wasn’t designed with the basic use case around technical computing. But I am sure that if you post in a Python forum that all of scientific Python is “fragile” because you can monkey patch np.dot to do something different, I’m sure the response would be “uhh… just don’t do that?”

7 Likes
83

I think this is probably the single clearest explanation of the ADL technical position so far and I wanted to highlight it as such. The explanation misses one fairly important aspect of the issue, however. Unlike single dispatch in C++ and other object-oriented languages, which by definition is dynamic, ADL is strictly a static name disambiguation done at compile time. In other words, in C++ there is always an equivalent program you could have written without ADL using fully qualified function names.¹ This is not a trivial difference and it means that suggesting that ADL is the natural extension of single dispatch to multiple arguments is highly misleading, if not flat out incorrect. Doing dynamic name resolution and dynamic dispatch on all of the arguments is not at all like C++’s ADL… it’s not like anything in any language that exists, which frankly gives me a great deal of pause. Julia already has one of the most powerful dynamic dispatch mechanisms anywhere; making it more complicated seems to pose a real danger of making it impossible to reason about and unusable for anything beyond toy programs.

¹ After template expansion: each expansion of templates can lead to ADL resolving a different function to call. This is still all static and done entirely at compile time, however.

I want to second this. I’m very glad to have understood the ADL perspective better. However, I’d like to clarify a few points:

  1. It was not new to us that single dispatch has a nice property in terms of avoiding putting method names into global namespaces. I’ve had dozens of conversations over many years with various core devs (including but not limited to Jeff) about this issue and various ways we could get the benefits of single dispatch for namespacing without messing up what is great about Julia’s multiple dispatch and generic function system. So far nothing has been a clear win.

  2. Technical problems are not the only thing preventing us from adopting ADL. ADL seems to be somewhat controversial even in C++ even though it is just syntactic sugar to avoid having to write out fully qualified function names. In Julia, you cannot statically resolve ADL because there are no static types by which to do so. Even if we came up with a dynamic analogue of ADL that did both dynamic argument-dependent function lookup and dynamic dispatch, I’m very far from convinced that it would be a good idea. At the risk of speaking for Jeff, I’m pretty sure he feels similarly.

  3. The goal of programming “without using” is not universal. In fact, I think the only person who has really enthusiastically espoused doing away with using is @jlperla. I understand why it’s appealing—less boilerplate is always nice—but I find the notion that you have no idea where methods you call might be coming from pretty unsettling. It works in C++ precisely because ADL is just a static rewrite to an equivalent fully qualified form. That fundamentally cannot be done in Julia since there are no static types. If we had “dynamic ADL without using” then the expression f(...) could, in general, call literally any method definition for the function name f in any code anywhere, even in code that I don’t use (even indirectly) and don’t know anything about. That is some seriously spooky action at a distance. In C++ I had to at least mention a type in my code that belongs to the namespace where the function definition lives. In Julia, there can be literally no connection between the caller and the method that gets called.

ADL is absolutely not the only approach to this issue and the fully unlimited “using-free” form of ADL that you support is definitely not the only option. There are several other ways to address this issue that have been discussed for Julia in the past. One of them is making the syntax x.f(y) mean something like moduleof(x).f(x, y). Another one is to provide better support for explicit generic function merging. Another would be automatic function merging when different functions with the same name are brought into scope by using. Yet another is deciding that all of the cures are worse than the malady. It’s a one-time annoyance to import the function you want to call, but it’s hardly the usability disaster that some have tried to make it out to be. Every paradigm has it’s strengths and weaknesses. Perhaps managing global names is just one of the things that’s a bit annoying in multiple dispatch.

16 Likes
84

That’s part of the functionality that I’d like to have available via @api that I’m working on, as well as renaming on importing.

85

7 posts were split to a new topic: Splitting more things out of base

86

Thank you. I think this is a very clear description of the technical implementation issue, and how the analogy might break down.

Yes, that is a very legitimate question of implementation. I bring up a using-free style partially because it is a mental test of namespace flexibility. But the main reason is that the advice people keep giving on discourse (“stop using using and import your functions”) contradicts with the reality that operators (and some functions?) are unusable without a using. There needs to be some education on how these things can be reconciled.

You are right, my suggestion was that ADL is the candidate for a general extension of semantics to capture a lot of related issues. And based on your statement about dynamic vs. static resolution of names, it could be simply that ADL is not possible here. Smaller issues can be dealt with and maybe it is enough if there are clear macros to merge methods, which are clearly documented and not something that requires constant arguments on.

But enough on ADL feasibility and desirability. Is there any appetite for compiling a summary of namespace differences from other languages for the documentation (and perhaps looking at a simple macro or two as part of the documentation for explicit function merging?)

2 Likes
87

I still don’t get this. In what way are operators and functions unusable without using?

2 Likes
88

It is exactly what is done in GAP, at least.

89

This together with a dialect like using except ... looks perfect to me.

1 Like
91

I think Jeff has hit the nail on the head here: the original proposal in this thread (which has been derailed with more ADL discussion) is fundamentally about dynamic scoping. In essence, it seems that in the proposal, which function is called is dynamically scoped (other variables remain lexically scoped, however, which is a bit inconsistent, but it’s just a proposal, so let’s go with it). Historically, dynamic scope was used in early Lisps, but it has largely fallen out of favor. The reason is precisely the objection that’s been given to this proposal: it makes it impossible to reason locally about the meaning of code since the meaning depends on the caller, which is not statically knowable.

I want to address the three issues that this proposal purports to solve:

Type piracy. Yes, it’s usually bad if you redefine what + on integers means. But I have not seen this to be a very large problem. People do it once at the REPL, Julia crashes almost immediately, and then they don’t do it again. We’ve contemplated having a way to seal certain functions or subsets of a function’s signature to prevent this, but it’s just not a high priority since it’s not a real problem in the wild. And sometimes type piracy is just fine and even useful. The notion that it’s an absolute evil is not something that the Julia core team has promulgated. We also didn’t coin the phrase “type piracy” although I do find it delightful.

Conflicting names. I think most of the ADL discussion applies here, so I’m not going to rehash it. Everything I said at the end of this post applies. Conflicting names can be an annoyance, but it’s not the end of the world. In case of conflict, be explicit: import or qualify. There’s no free lunch.

Binary compilation. Yes, Julia currently has a compile time problem. And yes, we very much need to be able to cache machine code in .ji files. But neither of these things have anything to do with how method extension works. The former is mostly a problem with LLVM getting a bit obese over the years and us adding more and more sophisticated optimizations—which everyone loves but which don’t come for free. The latter is just a matter of doing the work, which not very many people are able to do—and the ones that can have been busy getting 1.0 ready.

There are many features of Julia which cause us to generate a lot of code—aggressive specialization on argument types, specialization on function values, lots of inlining, implicit templatization of parametric types—but the way multiple dispatch and method extension work is not one of them.

8 Likes
92

I completely agree that this does it for (at least) a major version, and a using except can wait for a minor one.

Auto-merging might be nice, but in the short run even a documented explicit macro in Base for merging is a step forward (e.g. call @forcemerge solve(x::MyType) = 1 would do a merge into whatever solve currently exists, if required, and a @force import as in the starting point of GitHub - chakravala/ForceImport.jl: Macro that force imports conflicting methods in modules

That would be very easy to teach, and you wouldn’t need a bunch of fragile if VERSION <... like you do in https://github.com/wbhart/Nemo.jl/blob/master/src/Nemo.jl when Base decides it wants to change function names.

93

This seems to conflict with your earlier statement that GAP does not have namespaces.

94

That would require that except is a keyword in v1.0 since otherwise reclaiming it would be a breaking change.

By definition, this couldn’t happen until v2.0 since any function name change would be breaking change. This kind of thing is a pre-v1.0 issue.

95

I don’t think it’s currently a valid syntax so it would be possible. I don’t really see what the except buys you though. How is an explicit exception better than an explicit import elsewhere?

96

Might be worth it.

Aren’t all of the standard libraries (e.g. linear algebra, etc.) split off of Base in v0.7 allowed to change between major versions, aren’t they? If so, then adding/removing functions and operators from those (which would frequently be done with a using) could break code. Anyways, a way to easily force merges (macro for now) is the key. But this definitely is off topic of the original post.

97

The problem is with large modules which export a lot of things, including one or two which will conflict with another module. I don’t think you can do

using AModule

for everything and then use import only for the conflicting name. Once there is a conflict, you have to explicitly import everything. (Am I perhaps misinterpreting the rules here? At least this is what my experiments tell me. There is no error, but there’s always a warning.) That is a hassle, and that is why I longed for using except.

module AModule
export fun, notfun
function fun()
    println("AModule fun")
end
function notfun()
    println("AModule notfun")
end
end

module BModule
export fun, alwaysfun
function fun()
    println("BModule fun")
end
function alwaysfun()
    println("BModule alwaysfun")
end
end

using .AModule
fun()
using .BModule
BModule.fun()
alwaysfun()

gives

julia> include("test/playground.jl")
AModule fun
WARNING: using BModule.fun in module Main conflicts with an existing identifier.
BModule fun
BModule alwaysfun

EDIT: THIS IS A NON-ISSUE. SEE BELOW IN STEFAN’S EXPLANATION.

1 Like
98

Well, not having namespaces means automatic merging everything (like if everybody worked in Base).
This would be unmanageable without the mechanism to protect things from redefinition unless explicitly
overriden. On the other hand, my habits do not come only from GAP since in my life I used more than 30 programming languages (including C++ and lisp through macsyma). I like namespaces when they don’t get in my way. And there are things which I don’t like in GAP which are nice in Julia, starting with the possibility
to compile to efficient code which solves the 2-language problem.

99

If you have 50 symbols exported from a package that you want to use, but one of them conflicts with a name you are already using (and maybe can’t change), it’s much better to be able to import all of the names except for one, than have to explicitly import 49 symbols.

This, and being able to rename one or more functions on import in an easy fashion (again, to avoid a conflict) would also relieve some of the pain of the current system.

3 Likes
100

Is all of this stemming from people not realizing that you can do multiple using and import statements from the same module? For example:

julia> module A
           export a, b, c
           a = b = c = 1
       end
Main.A

julia> module B
           export c, d, e
           c = d = e = 2
       end
Main.B

julia> module C
           using ..A
           using ..B
           import ..B: c
           @show a, b, c, d, e
       end
(a, b, c, d, e) = (1, 1, 2, 2, 2)
Main.C

I only needed to do an explicit import for the one name that conflicts, c.

8 Likes
101

How come I get a warning? (Julep: Taking multiple dispatch,export,import,binary compilation seriously - #97 by PetrKryslUCSD)