Julep: Taking multiple dispatch,export,import,binary compilation seriously

Interesting points.

I have also been struggling to see how widespread binary compilation could be possible while keeping method overloading as flexible as it currently is. Its my biggest concern and uncertainty about Juia in the long term, although still really at an instinctual more than rational level.

The time and amount of restarts required, even using Revise, are the most painful thing about working in Julia (almost the only thing). I don’t understand the full potential for compiler optimisation, but it doesn’t seem like enough to resolve restart time without a cleaner binary cache.

Although I would really like to understand why not, if this is not the case!

2 Likes

I have tremendous respect for the posters on this thread, but I have to say that I also find the responses overly dismissive.

2 Likes

I’ve been reorganizing my strings code around a new principle, which is to have people pull in the names they want, via metaprogramming fun, instead of depending on what I also see as an overly simplistic model of export / import / using.

I just started this, so I haven’t fleshed out all the possibilities, but I think that one could achieve a lot of what @TsurHerman wants with no changes to the language.

If anybody is interested, please check out APITools (which I hope will get registered today :grinning: )

1 Like

My advice would be to look to the possibilities that the power of Julia’s metaprogramming can give you.
In the past, when I’ve had serious frustrations with the language, it’s frequently been able to come to the rescue (although with a serious amount of work, because my macro fu is not so great :wink:) and allow me to craft a nice alternative (such as adding Swift style interpolation and hex/Unicode constants, Python and C style formatting, and REPL style LaTeX / Emoji entities, in addition to HTML and Unicode entities, all to string literals).

Assume you are actually open to criticism, every single issues I’ve brought up in Possibility of `local import` statements in future? still hold. Namely,

  1. The additional dispatch rule complexity can only work if it essentially makes it the same as the dispatch rule now, i.e. even though it may not see all the definitions on all methods, it has to see all the definitions on the methods it use.
  2. It’s solving none issue. In fact, it makes binary compilation completely useless. The toplevel module is basically always user code (or even if it is not what you mean by toplevel, having to resolve the function at this level is at least something you have to support and not something that compilation can possibly handle) so you’ll have to compile everything when the user calls it.

Just to be more concrete, I’ll copy my example over, with changes so that the implementation details are hidden,

# All modules are using all other modules that the module is aware of (either because it's the user's choise to use them together or because it's an implementation detail) so no new using/import can be added.
module IfaceDefAndUse
# This module cannot be aware of any other modules
export g, f
function g end
f(a, b) = g(a, b)
end

module WrapperOfIfaceUse
using IfaceDefAndUse
export k
k(a, b) = IfaceDefAndUse.f(a, b)
end

module IfaceImpl
using IfaceDefAndUse
export T, f
struct T end
IfaceDefAndUse.f(::T, ::T) = 1
end

module Factory
using IfaceImpl

export l
l() = IfaceImpl.T()
end

module User
using Factory
using WrapperOfIfaceUse

m() = WrapperOfIfaceUse.k(Factory.l(), Factory.l())
m()
end

Now in the call chain, User.mWrapperOfIfaceUse.kIfaceDefAndUse.f no one is aware of IfaceImpl since that is an implementation detail as far as the user, who put them together, is concerned.

1 Like

In my example, B isn’t calling A.start directly, it is calling Base.sum, and Base.sum (which knows nothing about A or B) is calling Base.start (which now needs to call A’s version if you want sum to be generic). As @yuyichao says, the only way to make this kind of thing work is if Base.sum “knows” about all start methods including A.start, which leads you back to essentially the current dispatch rules.

You are proposing vast changes in the language on the eve of Julia 1.0. There was just a long thread explaining why a closely related proposal is unworkable. We’ve told you that such changes are not needed to enable static compilation in the future, and that in fact static compilation has already been proved to work in other languages with similar generic-function semantics (and already happens in the Julia sysimg). You haven’t worked through even an example as simple as Base.sum — and despite decades of study on the semantics of generic functions in multiple-dispatch languages, you haven’t pointed to any work that models the semantics that you want. You begin the thread by suggesting that we don’t currently take dispatch “seriously.” Is it so surprising that we don’t want to spend a lot of time debating you, trying to convert your proposal into a concrete algorithm and arguing its weak points?

By “armchair design” I’m referring to the practice of suggesting far-reaching changes and asking others to work out the details and the implementation. This is rarely successful.

5 Likes

This is what I’ve been saying since last winter already, that’s also why I created ForceImport.jl, my goal was to have a “local import”, which people thought was a completely useless and silly concept. It’s actually quite important of a concept for the language, but at this point I was able to resolve my issues with it by the @force macro.

The workflow with that macro actually helped me improve the precompile performance by 2x since I was extending so many methods, and was now able to start with a clean-slate method table for each method, hugely cutting down on the binary cache.

So as far as I am concerned, it is already possible to achieve the local import effect and its advantages without making any huge changes.

And just to set it straight since this seems to be a major misconception. The binary compilation does NOT care at all about dispatch (thanks to Jameson)!

The only part of the compiler that is sensitive to dispatch is type inference (it’s not the only part dispatch happens but it’s the only part static dispatch, or dispatch in the compiler, happens). The mapping from inferred code all the way down to binary code is completely julia dispatch independent.

The inferred code is already included in the package precompilation and from the julia dispatch perspective the next step to binary code is trivial. The main if not the only reason binary compilation is not widely used for packages is the availability of linkers or in general getting the low level library generation and loading logic straight. It’s not actually trivial or easy to implement but it has absolutely nothing to do with dispatch.

4 Likes

I think you are misrepresenting things. What was said was that this did not require a change in the language itself, nor that it required to have some extra support in Base. You got a lot of guidance on how to implement this and where to look for previous implementations of similar things. Taking the position that no one is listening to your great ideas is a bit unfair.

2 Likes

And I do appreciate all the guidance and support that was given recently, but I was initially ridiculed for it, so that memory had a lasting effect on me.

2 Likes

No one thinks that namespaces are useless and silly. And, of course, Julia already supported methods like Foo.+ that shadow (rather than extend) Base.+, and already supported importing them — that is what namespaces and other local scopes are all about. Your ForceImport package provides some syntactic sugar to import shadowed methods implicitly rather than explicitly, which no doubt is convenient in some cases, but does not change the dispatch or namespace semantics.

The problem comes if you want to extend (not just replace) Base.+, but only in a local context. That is what you initially said you wanted, and that is what the current thread tried to propose also. The difficulty arises because the whole point of extending (not replacing) a method is that one wants to extend the behavior of functions like Base.sum that call + outside the context of your local module. This is what is problematic, represents a complete upheaval of the language if it is even possible (without ending back at something equivalent to the current semantics), and is not provided by ForceImport.

9 Likes

I asked about being able to do that when I first started with Julia (i.e. extending a function in a way that only affects the function within the current module).
Since then, I’ve learned how to make a local definition and manually make it use the types local to the module, and otherwise fall back to the external (usually Base) definition.
I do wish that there had been some syntactic sugar for doing that, so now I’m looking at having macros take care of it for me, as much as possible, but I do agree with a number of the other people here, that there doesn’t need to be any real changes to the current dispatching model achieve that.

1 Like

Yup. The current approach is simply unable to cope with what many people (myself included) have been asking for, which makes it clear that it is only possible in Julia v2.0. This whole thread is yet another case of the inherent issues with the current lookup methods of functions and methods with namespaces. Maybe syntactic sugar is helpful to “get by” while new namespace/lookup design approaches are thought through, but the solution should not be to have a bunch of macros to muck around with other namespaces and manually merge what should be seamless.

Something as radical as Function name conflict: ADL / function merging? is the only way to fundamentally solve this (assuming that it works in Juilia). I think that we made progress there on understanding why the current design is surprising to many people from other languages, and how a using-free design was possible with argument dependent lookup.

But lets say that there is agreement with the core developers that there is a real problem and that lookup rules need to be reexamined (which I am not sure is true we have agreed on yet). Then what is the point of talking about this much pre-v2.0 design?

My answer is that if there is at least some agreement on the direction of method lookup in v2.0 (or that it should be reexamined) then it tells us what sorts of hacky macros are acceptable in the meantime. Put another way, if the end-result is that functions will always be in a global namespace - which counterintuitively ends up helping keep methods in local namespaces - then macro hacks which automatically merge methods are pretty innocuous.

My take is that Julia metaprogramming is powerful enough, that between now and when v2.0 starts being designed, a lot of things can be experimented with (as ideas of how to handle traits have been with Mauro’s packages), so that hopefully some consensus can be reached. (The consensus might just be, as in the case of @enum, that the “hacky macro” approach is just fine :grinning: )

It is the opposite. From what I can have seen, all people who have actually taken part in creating the julia language are vehemently opposed to anything resembling what is proposed here and in other threads. So to be blunt, this will not happen.

If this is a deal breaker, there are two productive ways forward:

  • Fork Julia, implement the suggested semantics and win users over by being so much better at dispatch, compilation, taking things seriously etc.
  • Find another language that already has these semantics implemented and hope for a better experience there.
4 Likes

Or Cassette’s context-specific dispatching could likely make it easy to come up with implementations of these kinds of things.

The best solution is to just not do type-piracy since if you avoid it then you won’t have any problems with method extension. Even if type-piracy didn’t effect other modules, having an overload on basic types that only happens in one module would result in very difficult to understand code anyways (it’s essentially macros to the extreme: all functions can have local contextualizations! Nothing is safe and nothing is marked as being different!), so “don’t do type-piracy” is enforcing an already good programming standard.

I also don’t see how this is solving any real problem other than letting people get away more easily with type-piracy. Method extension is a useful feature because it allows you to define actions on types in a standard way so that generic codes “just work”. @stevengj mentions that you extend Base.+ so that way Base.sum “just works”, but it goes even further: things which have actions like a number can work in DiffEq, Optim.jl, NLsolve.jl, etc. Types and their interfaces are defined via their actions. You cannot write generic code without an assumption of how certain actions work. This just obfuscates these issues to try to make programming look nicer, but results in the loss of one of Julia’s best features: the ease of composition of generic code on user-defined types.

If you’re not extending for the purpose of a useful implementation of an interface, then what for? To not use standardized syntax and apply names/functions in unconventional ways specifically in a single module/script? While I agree that it’s everyone’s right to write bad code, I fail to see how a feature how code obfuscation for mitigation of downstream effects of type-piracy would be a high priority. I would think pretty much everyone working on Julia or using Julia sees other things (caching of native code, debuggers, closure issues, the new optimizer / compilation times, Pkg3, improved linear algebra via BLIS, machine learning / database / data frame/ diffeq / optimization libraries, etc.) as higher priority, which is why this falls on deaf ears.

9 Likes

I’ll have to look into that.

Lately though I’m really sorely tempted to do some type-piracy, to be able to fix some bugs and limitations in Base just by loading my package :wink: (Regex, I’m looking at you!)

This is one of the least helpful comments I have ever seen on discourse, and certainly not the right way to talk to users and people working hard to push the language. It is perfectly reasonable to say “dealing with potential 2.0 design from armchair language designers is low-priority”, but this response is a about the worst thing you could say to me right now. I was seriously consider rethinking your approach to engaging in legitimate discussions and issues with Julia (i.e. it is impossible in julia right now to write practical code with using, and it is possibly fixable).

Luckily I didn’t see this sort of tone when discussing the details with other core developers in Function name conflict: ADL / function merging? when we finally broke through in understanding what the fundamental naming issue is. And for some reason I suspect you have not digested what we discussed there and why Julia is so confusing to people coming from every language except lisp.

2 Likes

With a decoupling of function and method namespaces, type-piracy is a thing of the past. It strengthens the ability to put everything (including operators) in your own namespace for your own types. It is the only practical way to write code which has not using at all, even using Base.

It was not meant to be helpful, it was meant to be truthful. What you want to do with that information is up to you.

I think you should speak for yourself here. I find it very possible to write practical code in julia.

The people I have introduced to julia has given me the impression that they think the multiple dispatch system is very clear and intuitive and have not coded a single line of lisp in their lives. Neither have I for that matter.

2 Likes