Julep: Taking multiple dispatch,export,import,binary compilation seriously

In my example, B isn’t calling A.start directly, it is calling Base.sum, and Base.sum (which knows nothing about A or B) is calling Base.start (which now needs to call A’s version if you want sum to be generic). As @yuyichao says, the only way to make this kind of thing work is if Base.sum “knows” about all start methods including A.start, which leads you back to essentially the current dispatch rules.

You are proposing vast changes in the language on the eve of Julia 1.0. There was just a long thread explaining why a closely related proposal is unworkable. We’ve told you that such changes are not needed to enable static compilation in the future, and that in fact static compilation has already been proved to work in other languages with similar generic-function semantics (and already happens in the Julia sysimg). You haven’t worked through even an example as simple as Base.sum — and despite decades of study on the semantics of generic functions in multiple-dispatch languages, you haven’t pointed to any work that models the semantics that you want. You begin the thread by suggesting that we don’t currently take dispatch “seriously.” Is it so surprising that we don’t want to spend a lot of time debating you, trying to convert your proposal into a concrete algorithm and arguing its weak points?

By “armchair design” I’m referring to the practice of suggesting far-reaching changes and asking others to work out the details and the implementation. This is rarely successful.

5 Likes

This is what I’ve been saying since last winter already, that’s also why I created ForceImport.jl, my goal was to have a “local import”, which people thought was a completely useless and silly concept. It’s actually quite important of a concept for the language, but at this point I was able to resolve my issues with it by the @force macro.

The workflow with that macro actually helped me improve the precompile performance by 2x since I was extending so many methods, and was now able to start with a clean-slate method table for each method, hugely cutting down on the binary cache.

So as far as I am concerned, it is already possible to achieve the local import effect and its advantages without making any huge changes.

And just to set it straight since this seems to be a major misconception. The binary compilation does NOT care at all about dispatch (thanks to Jameson)!

The only part of the compiler that is sensitive to dispatch is type inference (it’s not the only part dispatch happens but it’s the only part static dispatch, or dispatch in the compiler, happens). The mapping from inferred code all the way down to binary code is completely julia dispatch independent.

The inferred code is already included in the package precompilation and from the julia dispatch perspective the next step to binary code is trivial. The main if not the only reason binary compilation is not widely used for packages is the availability of linkers or in general getting the low level library generation and loading logic straight. It’s not actually trivial or easy to implement but it has absolutely nothing to do with dispatch.

4 Likes

I think you are misrepresenting things. What was said was that this did not require a change in the language itself, nor that it required to have some extra support in Base. You got a lot of guidance on how to implement this and where to look for previous implementations of similar things. Taking the position that no one is listening to your great ideas is a bit unfair.

2 Likes

And I do appreciate all the guidance and support that was given recently, but I was initially ridiculed for it, so that memory had a lasting effect on me.

2 Likes

No one thinks that namespaces are useless and silly. And, of course, Julia already supported methods like Foo.+ that shadow (rather than extend) Base.+, and already supported importing them — that is what namespaces and other local scopes are all about. Your ForceImport package provides some syntactic sugar to import shadowed methods implicitly rather than explicitly, which no doubt is convenient in some cases, but does not change the dispatch or namespace semantics.

The problem comes if you want to extend (not just replace) Base.+, but only in a local context. That is what you initially said you wanted, and that is what the current thread tried to propose also. The difficulty arises because the whole point of extending (not replacing) a method is that one wants to extend the behavior of functions like Base.sum that call + outside the context of your local module. This is what is problematic, represents a complete upheaval of the language if it is even possible (without ending back at something equivalent to the current semantics), and is not provided by ForceImport.

9 Likes

I asked about being able to do that when I first started with Julia (i.e. extending a function in a way that only affects the function within the current module).
Since then, I’ve learned how to make a local definition and manually make it use the types local to the module, and otherwise fall back to the external (usually Base) definition.
I do wish that there had been some syntactic sugar for doing that, so now I’m looking at having macros take care of it for me, as much as possible, but I do agree with a number of the other people here, that there doesn’t need to be any real changes to the current dispatching model achieve that.

1 Like

Yup. The current approach is simply unable to cope with what many people (myself included) have been asking for, which makes it clear that it is only possible in Julia v2.0. This whole thread is yet another case of the inherent issues with the current lookup methods of functions and methods with namespaces. Maybe syntactic sugar is helpful to “get by” while new namespace/lookup design approaches are thought through, but the solution should not be to have a bunch of macros to muck around with other namespaces and manually merge what should be seamless.

Something as radical as Function name conflict: ADL / function merging? is the only way to fundamentally solve this (assuming that it works in Juilia). I think that we made progress there on understanding why the current design is surprising to many people from other languages, and how a using-free design was possible with argument dependent lookup.

But lets say that there is agreement with the core developers that there is a real problem and that lookup rules need to be reexamined (which I am not sure is true we have agreed on yet). Then what is the point of talking about this much pre-v2.0 design?

My answer is that if there is at least some agreement on the direction of method lookup in v2.0 (or that it should be reexamined) then it tells us what sorts of hacky macros are acceptable in the meantime. Put another way, if the end-result is that functions will always be in a global namespace - which counterintuitively ends up helping keep methods in local namespaces - then macro hacks which automatically merge methods are pretty innocuous.

My take is that Julia metaprogramming is powerful enough, that between now and when v2.0 starts being designed, a lot of things can be experimented with (as ideas of how to handle traits have been with Mauro’s packages), so that hopefully some consensus can be reached. (The consensus might just be, as in the case of @enum, that the “hacky macro” approach is just fine :grinning: )

It is the opposite. From what I can have seen, all people who have actually taken part in creating the julia language are vehemently opposed to anything resembling what is proposed here and in other threads. So to be blunt, this will not happen.

If this is a deal breaker, there are two productive ways forward:

  • Fork Julia, implement the suggested semantics and win users over by being so much better at dispatch, compilation, taking things seriously etc.
  • Find another language that already has these semantics implemented and hope for a better experience there.
4 Likes

Or Cassette’s context-specific dispatching could likely make it easy to come up with implementations of these kinds of things.

The best solution is to just not do type-piracy since if you avoid it then you won’t have any problems with method extension. Even if type-piracy didn’t effect other modules, having an overload on basic types that only happens in one module would result in very difficult to understand code anyways (it’s essentially macros to the extreme: all functions can have local contextualizations! Nothing is safe and nothing is marked as being different!), so “don’t do type-piracy” is enforcing an already good programming standard.

I also don’t see how this is solving any real problem other than letting people get away more easily with type-piracy. Method extension is a useful feature because it allows you to define actions on types in a standard way so that generic codes “just work”. @stevengj mentions that you extend Base.+ so that way Base.sum “just works”, but it goes even further: things which have actions like a number can work in DiffEq, Optim.jl, NLsolve.jl, etc. Types and their interfaces are defined via their actions. You cannot write generic code without an assumption of how certain actions work. This just obfuscates these issues to try to make programming look nicer, but results in the loss of one of Julia’s best features: the ease of composition of generic code on user-defined types.

If you’re not extending for the purpose of a useful implementation of an interface, then what for? To not use standardized syntax and apply names/functions in unconventional ways specifically in a single module/script? While I agree that it’s everyone’s right to write bad code, I fail to see how a feature how code obfuscation for mitigation of downstream effects of type-piracy would be a high priority. I would think pretty much everyone working on Julia or using Julia sees other things (caching of native code, debuggers, closure issues, the new optimizer / compilation times, Pkg3, improved linear algebra via BLIS, machine learning / database / data frame/ diffeq / optimization libraries, etc.) as higher priority, which is why this falls on deaf ears.

9 Likes

I’ll have to look into that.

Lately though I’m really sorely tempted to do some type-piracy, to be able to fix some bugs and limitations in Base just by loading my package :wink: (Regex, I’m looking at you!)

This is one of the least helpful comments I have ever seen on discourse, and certainly not the right way to talk to users and people working hard to push the language. It is perfectly reasonable to say “dealing with potential 2.0 design from armchair language designers is low-priority”, but this response is a about the worst thing you could say to me right now. I was seriously consider rethinking your approach to engaging in legitimate discussions and issues with Julia (i.e. it is impossible in julia right now to write practical code with using, and it is possibly fixable).

Luckily I didn’t see this sort of tone when discussing the details with other core developers in Function name conflict: ADL / function merging? when we finally broke through in understanding what the fundamental naming issue is. And for some reason I suspect you have not digested what we discussed there and why Julia is so confusing to people coming from every language except lisp.

2 Likes

With a decoupling of function and method namespaces, type-piracy is a thing of the past. It strengthens the ability to put everything (including operators) in your own namespace for your own types. It is the only practical way to write code which has not using at all, even using Base.

It was not meant to be helpful, it was meant to be truthful. What you want to do with that information is up to you.

I think you should speak for yourself here. I find it very possible to write practical code in julia.

The people I have introduced to julia has given me the impression that they think the multiple dispatch system is very clear and intuitive and have not coded a single line of lisp in their lives. Neither have I for that matter.

2 Likes

Sorry, meant to say “practical code without using”. But please reread that ADL chain, as I think you are applying an extremely dogmatic prior to this all.

This has absolutely nothing to do with julia having multiple-dispatch, it has to do with namespace lookup, which is largely orthogonal (even if it gets trickier).

But I don’t want to rehash that thread again, as it was documented where the confusion lies.

But then what does your code even mean or do? Now your code uses the same syntax as everyone else, but does different things than Base! Now your code adds a + function to your number type but then putting that number into Base.sum, DiffEq, Optim.jl, NLsolve.jl doesn’t work. Now your “generic functions” don’t work on number types you haven’t specifically programmed them for, so they aren’t generic at all. And if you want to just take over every single dispatch to reprogram someone else’s generic code… isn’t this just Cassette’s contextual dispatch?

We all know the things you cannot do if you want to avoid type-piracy. The fact that you cannot do those things doesn’t mean they are a good idea to do it in the first place.

1 Like

With multiple-dispatch I meant the whole dispatching system (including extending functions defined in other modules).

Anyway, carry on. I just wanted to bring expectations of the outcome here perhaps a bit closer to reality.

Sometimes code in Base itself commits type-piracy.
I’ve been running into problems with over-broad function deprecations.
My believe is that deprecations should only be for the specific methods that were present (i.e. what methods(function) shows).
If I’ve extended replace or find, perfectly legally, calling them only with my own types, without any type-piracy at all, and then deprecations.jl adds a deprecation that makes all find go to findall, that breaks all my code.

Just to be clear, did you only extend replace or find to make it convenient to call the functions? If so, this is the lack of ADL rearing its ugly head… Not to keep beating a dead horse, but if methods can live with their own types in practice, this stuff is never an issue.