How do you deal with generic functions calling interface method extended in another module.?
You mean like FileIO?
the interface function which get extended for dispatch would need to be a macro(which is appropriate because
it is a kind of magic … which is what macros do … from a programmer perspective) which calls:
function load(s::Stream{format"PNG"}; keywords...)
based on the extension.
Thus forcing the programmer to state each format she wants to register for use in this scope. but also frees her to choose an implementation for a specific file format.
EDIT:
not sure if it is possible with a macro … you are the expert, what do you say is it possible?
One of the other ways I am using macros to handle method dispatch is using Reduce switches,
julia> using Reduce
julia> @rounded @factor x^3-2x+1
:((x + 1.61803398875) * (x - 1) * (x - 0.61803398875))
In this example, any sequence of REDUCE switch macros can be used, and they will all be automatically rewritten into a single command to the software, instead of communicating multiple times.
@force using Algebra
However, some of the extensions currently available in the package will now be moved to the sub-module Algebra, but will still be optional to have the full extensions of all methods in any module with @force macro.
I don’t see what’s the difference between a macro and a method here. It’s not just FileIO, it’s everything or at least any abstract types. Custom number types, custom array types, etc. You just can’t require the caller of the function to know which module the method is going to come from. I do believe adding namespace lookup based on type in certain cases is good, however, it shouldn’t change how method/function/dispatch works.
What about the following inference rule, if M is a the calling module and B is the module in which the function being called is defined then any function call inside the called function is inferred first based on the local
scope (module B) as defined in my original post, and then based on the calling module scope(given that it is exported in the calling module).
The most concrete match is chosen, if there are 2 matches with have the same concreteness then the one from
module B(the function location) is chosen.
Module Main will be special in the sense that any function in Main is considered exported.
I believe this change in architecture is not far fetched or a drastic changed compared to what is going on with inference today.
I just believe it is better and clearer from the programmer perspective. giving some encapsulation to multiple dispatch
@jeff.bezanson @StefanKarpinski , If you have a few minutes spare I would highly appriciate your opinion on the matter.
I appologize if this idea is not perfectly articulated … and I would understand if you choose not to comment.
- It’s not inference, but dispatch.
- There’s nothing special about the caller’s module. Your call could be a few levels of function calls deep and easily have many modules in between.
- Making functions behavior sensitive to the callers’ scope will both make it impossible to pre-compile and very hard to understand.
2 Likes
To update on what’s happening with the implementation, I made a comparison of load times using the original form of extending the Base functions and the new way of containing them locally. There is performance gain.
On the original master branch, the compile cache is 2.5 MiB and the load time is
julia> @time using Reduce
Reduce (Free PSL version, revision 4588), 28-Apr-2018 ...
11.211845 seconds (7.25 M allocations: 402.106 MiB, 3.00% gc time)
While on the force-user brach, the compile cache is 1.7 MiB and the load time is
julia> @time using Reduce
Reduce (Free PSL version, revision 4588), 28-Apr-2018 ...
6.455135 seconds (3.57 M allocations: 191.129 MiB, 4.70% gc time)
This difference can be attributable to the fact that the package contains definitions for over 50 conflicting methods from Base, such as arithmetic, matrix, trigonometric, and other elementary / special functions.
With the new definitions, the original dispatch from Base is essentially reduced down to a single one that encapsulate all of the Base methods, plus the additional new ones defined by the package.
Now, the multiple-dispatch is handled completely differently, which is why the tests are failing. The multiple-dispatch had to be very finely tuned to be in harmony with Base, and there were no issues or conflicts due to the precise fine tuning of the dispatch. However, since a simpler dispatch is now used using the
module Algebra
export +
+(r...) = Base.:+(r...)
end
using that form extension causes havoc on the multiple dispatch, since it was so finely tuned to work with the hundreds of existing methods from Base. This means that they fine tuning of dispatch has to be started over, since none of the dispatch structure from Base could be copied over to the new module.
I disagree, the more I run it my head it is clearer to me that this is the right approach ,
Modules will be more encapsulated , importing packages will not alter the inner workings of other packages . It is easier to understand what gets export d and extended .
The conceptual difference is that dispatch is rooted at the current module and not in a common ancestor.
Actually this makes modules easier to compile (to binary not just into IR) because only exported functions will be “dynamic” whereas today as with the example I gave before I can import a function that will crush the whole system in an instant.
About the few layers deep of function calls … I fail to see how this is different or more complicated than what’s currently happening.
Maybe this idea is to advanced for you at the moment…
This is not very well defined and it’s the biggest problem. See below.
Right. But the point is that this makes everything useless. You cannot create any abstract interface anymore.
It won’t unless you remove your special treatement of module. It’ll make compiling strictly worse, by a lot.
There are very few general purpose systems that stops you from doing that in all possible ways, as long as you still want your program to be useful. As long as you don’t do things that are strongly recommended against (type piracy in this case), there won’t be a problem.
You were proposing making the caller module of a function special. However, the actual user of the function you want the callee to see may not be the caller of the function since there may be many layers of module in between. From what I can tell, you are saying that f(a, b) = a + b will behave differently if it’s implemented as f(a, b) = f2(a, b); f2(a, b) = a + b since the + in the second function won’t know which module f is being called anymore.
You are welcome to try a more advanced language or forum. If anything I talked about above is too advanced or at least too abstract for you, feel free to ask about the precise part that’s unclear.
3 Likes
you made a lot of assumptions regarding what I meant … instead of joining into the mental exercise of listing the steps that make this possible.
And no I do suggest all functions will depend on the caller module , just exported function or better yet “interface” functions who get their dispatching inference rules from the calling module (like macros which get evaluated at dispatch time scope and not where their code was defined). All other functions have their dispatch rules according to the module they were defined.
Do you have a self destruct example for Python? Ruby? Go? Rust? Something as simple as the one liner example I gave?
My wish is that instead of putting lots of effort of showing me were I am wrong … you would use your superior experience to join in and see what would actually make this a reality … not nessecearly a reality for Julia.
I personally think Julia is a brilliant concept , but it does not mean that a better and more concise way of doing things doesn’t not exist.
With respect, I don’t think this is how it works in the open source world. Specifically, inviting other people to use their time, effort, and expertise to work out the implementation for something that you are interested in but they are explicitly skeptical about is unlikely to lead to any results. Especially after suggesting
1 Like
More over , it seems to me that sometime in the past there was an actual flash of light coming out of Julia’s behind , some are still blinded by it mistaking it for the sun…
This is the most alarming thing about the current implementation…
Why is this alarming? That it is possible at all? You are running arbitrary code, so that it is possible to crash the program is not very surprising.
The fact that this is possible at all is a powerful feature of Julia that allows many cool things to be done. I would be sad to see that go.
1 Like
I agree that the language does cool things It would be even cooler to be able to do all of those in a way that is not so breakable.
I think multiple dispatch can be achieved in a way that keeps the language more or less as it is while offering safety and guarantees that if a function f in module M was compiled for some concrete types then… as long as you don’t change M or its imported packages … that binary holds.
What a I suggested is something along the lines of:
Drop the concept of extending and importing functions
A) imported functions having the same name gets fused into one function with several methods.
B)
resolve the right binary method to call in the current module … and in the calling frame … and in the calling frame of the calling frame … etc… all the way to the top and chose the most specialized and “near” implementation.
It will take me some time to formalize this …or go over the implications : I was hoping to go into discussion of … in order to do that you’ll need A B C etc
which in the end may turn out to be too complicated or … better than the current way dispatch works.
Can you please clarify which feature of the language you have a problem with? The ability to write methods
- for functions defined in another module,
- for functions and types defined in other modules (not necessarily the same),
- for functions and types, all defined in the same module,
- that overwrite existing methods, defined in another module?
Guidelines about type piracy are not formally enforced by the language, but catch (2) and (3), and also (4) if the other definition that would be overwritten avoids type piracy. OTOH, restricting (1) would kill a very useful feature.
AFAICT, what I’ve stated does not contradict with what you’ve said so all the assumptions that I (and everyone
reading what you’ve said must) made are everything that you failed to mention. In another word, it’s impossible to understand your full proposal without those assumptions.
As an example, I don’t believe you’ve mentioned the relation between exported function and the sensitivity to calling modules until this reply so there’s no way I could have guess that. (Not that this assumption actually matter though, see below.)
By all mean, the ONLY way to push such an idea forward is to identify why it won’t work and then how it can be fixed. Just asking others to admire what you’ve proposed will not lead to anywhere. For a big change like what you’ve proposed, it’s very important to figure out what impact does it have. In particular, how does it affect usecases that we must support. Identifying how you want to support these usecases and avoiding seemingly obvious issues is also one of the best ways to understand all the non-obvious detail of your proposal.
As far as what I can tell, this does not make a difference and all what I’ve said about my f and f2 still apply. In this case, f is the exported function and f2 is “all other functions”. This basically forbid one from using these “all other functions”.
What you have actually does NOT crash julia. What it crashes is the REPL. All core components are actually well protected by that.
This is exactly my point, if we handle dispatch differently you could be restricting (1) without loosing
what is (positively)enabled by it.
I am afraid I don’t see the details of how your proposed mechanism would work. Perhaps you could open a Julep which describes it in detail, considering its impact on all current use cases.
3 Likes
I am not in it for admiration, but for open mindedness ,learning ,the joy of solving abstract stuff together.
And you disappoint me because you are obviously capable and probably better at this stuff (with respect to Julia internals) than me, and yet you too easily regard any idea that is different than the status quo as being wrong.
Anyway I did come up to a very very preliminary synopsis of the idea , eventually, in this discussion.
it goes like this:
dispatch can be defined recursively as follows: for a function F who is being called from function G
resolve the most appropriate method to call in the Module containing F, and also the most appropriate method
in the Module containing G if and only if F is exported (not sure this restriction is needed…) ,choose the most specialised method.
if there is no G then selection is trivial.
module IterInterface
start(iter) = throw("unimplemented")
next(iter, state) = throw("unimplemented")
done(iter,state) = throw("unimplemented")
export start,next,done #actual definition of the interface is redundant
end
module ForLoop
using IterInterface #for formality .. not really needed
forloop(f::Function,iter) = begin
state = start(iter)
while !done(iter,state)
elem,state = next(iter,state)
f(elem)
end
end
export forloop
end
module Foos
struct Foo
v
end
start(iter::Foo) = false
next(iter::Foo, state) = (iter.v,true)
done(iter::Foo, state) = state
export Foo,start,next,done
end
module Main
using Foos
using ForLoop
forloop(println,Foo(10))
end
in this example the start(iter) method called from forloop which is called from Module Main will resolve correctly.