Julep: Taking multiple dispatch,export,import,binary compilation seriously

[ChrisRackauckas]

Here’s my summary of what’s going on in this thread and the other. Take it or leave it. I wanted to write this down so that way people can understand that the responses are not dismissive, but clearly identify the reasons why no one is really gung-ho on implementing this.

Motivation

It starts with some assumption that Julia’s current function namespacing is hard to understand for people coming from single dispatch languages. Example statements:

@jlperla

I have tried to say that everyone has been confused on this topic, myself likely the most confused of all through most of it (though I think I finally “get it”). I had suggested from your previous summary of the ADL thread (which talked about multi-dispatch) that you stopped reading and not fully digested the conversation, but I apologize if you had.

There’s no objective data on this, just some people saying “yes I found it confusing” (and others not being confused about it, I’m in the latter camp). It’s normally people who are less familiar with Julia dev saying it’s confusing (which is the where the “contentious split” comes from). This then leads to the conclusion by some that further namespacing of functions to allow for extension in new ways is a feature/design that should be worked towards. However, there’s two objections to that.

Objection 1: Utility

First of all, there’s the objection about whether it’s actually better. @jeff.bezanson and I have argued that the resulting code from having multiple functions merge on their own could be more non-local and more confusing. For example:

@jeff.bezanson

This is closely related to the notion of “dynamic scope” (as opposed to lexical scope), where variables are looked up in calling scopes. A general problem with such approaches is that they make it hard to reason about code. If I write

f() = sum([1,2,3])

in my package, I intend for f() to return 6 . In your proposal, unless I misunderstand it, I have no idea what f does. If it’s called from a package that defines sum(x) = 42 , or a+b = 42 , then f() will return 42 . Yes, currently it’s possible to overwrite Base.sum or + of integers, but we discourage that and print a warning if you do it. But in your proposal it would just be the correct behavior to let another package replace the sum or + you intended to call.

This is the same thing that I was saying earlier with

@ChrisRackauckas

But then what does your code even mean or do? Now your code uses the same syntax as everyone else, but does different things than Base! Now your code adds a + function to your number type but then putting that number into Base.sum , DiffEq, Optim.jl, NLsolve.jl doesn’t work. Now your “generic functions” don’t work on number types you haven’t specifically programmed them for, so they aren’t generic at all. And if you want to just take over every single dispatch to reprogram someone else’s generic code… isn’t this just Cassette’s contextual dispatch?

So essentially the objection is that the issues you can get from extending functions is really due to abusing existing names and writing something confusing in the first place. I haven’t seen a counter to that in this thread.

Objection 2: Implementation Feasibility

@stevengj, @yuyichao, @mbauman, etc. have engaged in more technical discussions about implementability of these features. While @jlperla brings up here and in Function name conflict: ADL / function merging? that C++ does something similar to what’s proposed, the issue is that there doesn’t seem to be another language that actually achieves this with first-class functions and multiple dispatch. Putting all of those features together is the combination that is difficult to handle. For example:

@jeff.bezanson

The key phrase here is “for a given function”. The entire problem is how to identify which function you’re talking about. For example, if a function is just identified by a name (as it is in class-based OO languages) then the rest of your proposal goes through no problem—we could implement it e.g. by having a single namespace of functions.

In Julia, a “function” can be any type with a call added to it. Functions are not distinct from other variables, and names can be dynamically changed since Julia is dynamic. So the implementation has to account for this. So the problem is that @stevengj’s question:

@stevengj

For example, think about what happens when a module A defines a new type T and extends the iterator functions Base.start etcetera to work on T. Then module B calls module A to create a x = T() and then calls Base.sum(x) , which calls Base.start(x) etcetera — this works because Base.start has been extended to work on T , regardless of where it is called from. How does Base.sum call A’s iterator methods if A.start and Base.start are entirely distinct?

while it’s answered by the simple “it’s easy, C++ does it!” doesn’t necessarily apply to Julia. To keep the nice features of Julia (which are essential to generic programming in this language) and have this kind of merging is completely unknown, and it’s unknown whether it’s feasible. This is why the core devs have proposed technical questions and have asked for PRs / prototypes because it’s unclear how this could even be done. If someone thinks it’s possible, the best way to answer these technical questions is to have a working code that doesn’t have these issues but keeps Julia’s core features in tact.

Summary

So it’s obviously not making it into Julia v1.0. But it’s also not dismissive to say that it’s not a v2.0 priority either. Julia is not MATLAB/Python/R/C++/etc., it’s different in many ways. The way that Julia does generic programming is relatively simple and dynamic which is quite unique. Trying to bring Julia in line with the other languages should not drop this core element of Julia’s design and usage. Objection 1 is basically noting that these changes don’t necessarily help generic programming in Julia. Objection 2 is noting that coming up with an implementation that doesn’t effect Julia’s generic programming structures is really hard if not impossible. The fact that it’s hard to do and doesn’t have a clear positive outcome is why people aren’t throwing their free time on this project.

The Evolution of the Julia Community

I think this needs to be ended with a statement on the current state of the Julia community. To outsiders and newcomers, Julia looks new because you just found it. It’s a language where things are being experimented on, it’s a language which is hot on the “what’s new to learn” lists, etc. People jump in thinking there’s huge wholes in the package ecosystem that need to be filled and it’s the wild west. People call it “in beta” since it’s not v1.0.

However, Julia is definitely not in beta and it’s well into its transition into “not new”. There is an established community. There are large state-of-the-art Julia packages/projects (JuMP, DifferentialEquations.jl, Flux.jl, etc.) which make heavy use of Julia’s unique features. There’s common styles, learning materials, etc. There are people hired to work on and with the language. The language itself is gearing up to release its 1.0 long(er)-term support release in just a few months. And for reference, the language was publicly released more than 5 years ago.

Over these last five years, people have built impressive software using the existing language. It’s popularity routinely lists somewhere around things like Haskell and Lisp. There is no question about whether Julia “has made” or will “stick around”: Julia has already made it and has a large enough community to keep it around, not to mention a stronger funding structure and a larger dev community than any of the other open source languages I’ve seen.

This is where the disconnect seems to happen in these kinds of threads. Newcomers tend to make statements like “it will only be adopted if …”, but everyone around here adopted Julia along time ago and is using Julia not in spite of these differences, but because of these differences. If you think about the issue like that, it’s probably clear why arguments based on “but I know xyz MATLAB/R/Python people won’t adopt Julia until …” is met with “who cares? Julia is for us because we think it’s better, and we’re willing to share it but don’t want to lose what makes Julia unique”. Yes, Julia isn’t as popular as Python, and we can argue endlessly about how to market better (and whether it’s actually a good thing to be that popular at the cost of being so generic, I’m not entirely convinced that’s a good thing). However, arguments about changing Julia will do much better if it’s not discussed like a shiny new toy to play around with, and instead is discussed as the production-ready software that it is.

This has to mentioned because these kinds of threads have a heavy premise in their motivation that for Julia to be “useful” it may need to adopt xyz from some other language to either have people transition or to implement some feature correctly. But this assumption is wrong. People already have and continue to use/pickup Julia because of what it currently does. You can not argue that the Julia should adopt what everyone else is doing without discussing whether it’s better than what Julia is already doing.

Me?

As a specific example for this case, DifferentialEquaitons.jl has worked extremely well because of how Julia’s generic programming works. I am not sure how possible this would’ve been in MATLAB/Python/R/C++/etc., at least without substantial amounts of extra work. Maybe there is a way to work out all of this extra namespacing + auto-merging in a way that doesn’t effect the current generic programming structure at all, who knows. But you aren’t going to convince me that Python does generic algorithms better (it doesn’t), and so if you don’t have a way that keeps Julia’s generic functions over abstract number/array types in tact, I don’t want it. If your proposal does a substantial break to this (and other) existing code then it’s not necessarily a good thing unless I have some very tangible benefits. It’s not dismissive to say “if you want the Python way, then go use Python” because they are different for a reason. I have used both and think Julia’s is much better for building exactly the things I have built.

This is long but hopefully puts everyone on the same page. The technical questions and the community statements are heavily intertwined in the discussion, so I hope this helps parse out why those more established seem to be “against” the idea and why this easily raised tensions.

Cheers.