So basically you treat all non-techincal admiration as proof that what you are doing is in the right direction and you treat all detailed technical explaination as “just your opinion”. I see the difference between “criticise freely” and actually accepting critisism now.
Now back to the more techinical part.
Shared libraries are useful because they are compiled AHEAD OF TIME. How many times something is compiled does NOT matter. For what it worth, a specialization of a method in a certain world range will already only be compiled once (ok if it’s inlined that’s slightly different but the whole point of inlining is to compile multiple times). What you propose will won’t help even compiling things once since
There isn’t any problem of compiling multiple times to begin with so nothing can help.
Everytime you imported something the whole lookup will need to be redone, exactly the same as what it is now.
Absolutely not. What you are saying is that there should be no name isolation, no implementation detail. All the implementation detail has to be exported and defeating what you are trying to do anyway.
Except that you don’t want sum to call Base.start or B.start, you want to it to call A.start (because you are summing an A.T() object, even though you are not calling from module A.)
(If your answer is that the A.start method was merged with B.start and Base.start when B imported A, then you’ve arrived essentially back at the current semantics: if you import module Aanywhere in your program, then A.start needs to be visible everywhere in your program, because an x = A.T() object could be passed to any scope in your program, and that scope could call e.g. Base.sum(x).)
@jrevels talk last night on Cassette.jl was great (it seems so magical!) and Jeff had some interesting ideas on how to use it for debugging (which was great news, since Traceur.jl is broken on master (not it’s fault, ASTInterpreter2.jl doesn’t load on master).
I’m going to investigate Cassette later today)
If I understand correctly, you’re proposing looking at the call stack and using the module of the calling code. If so, mentioning the “AST” here is incorrect, as the call stack is a dynamic property, not a static/syntactic property. You cannot determine the caller of a function by looking at ASTs.
This is closely related to the notion of “dynamic scope” (as opposed to lexical scope), where variables are looked up in calling scopes. A general problem with such approaches is that they make it hard to reason about code. If I write
f() = sum([1,2,3])
in my package, I intend for f() to return 6. In your proposal, unless I misunderstand it, I have no idea what f does. If it’s called from a package that defines sum(x) = 42, or a+b = 42, then f() will return 42. Yes, currently it’s possible to overwrite Base.sum or + of integers, but we discourage that and print a warning if you do it. But in your proposal it would just be the correct behavior to let another package replace the sum or + you intended to call.
This proposal also exposes implementation details. In the example in your second post, M needs to add using StaticArraysbecause N uses static arrays. That means if N decides to change its implementation to use StaticArrays, that breaks all of its users. All of its users need to add using StaticArrays in order to work again. I believe that is unacceptably non-modular and makes this proposal unworkable.
Also python, object-oriented matlab, anything with single-dispatch or operator overloading, etc. Basically almost any other language users will come from. That isn’t to say that the way Julia has done it is wrong, just that unless we can all find a way to tightly communicate the underlying mechanics this will keep coming up. People are not used to the power (and consequences) of first-class functions, even if they have dealt with function overloading and dynamic dispatching.
Yes, this is exactly what needs to be communicated! The way Julia has done namespaces is the natural way to add them to a language with first-class functions… although I don’t know how many languages with first class functions have multiple-dispatch and namespaces. It seems that most languages with first-class functions avoid namespaces, or have similar issues far worse than Julia (e.g. Anonymous records. A solution to the problems of record-system. · GitHub )
The way that someone thinking about extending multiple-dispatch as a generalization of single-dispatch is different. Are they reconcilable? No one knows. Is an implementation feasible? Not sure, and it isn’t the time to find out. ADL is the only hope for something that would extend the current semantics and let people better segment namespaces of methods (essentially by have no namespaces for functions). Those questions can be pushed off for now.
If that is your impression, then it is mistaken. I have tried to say that everyone has been confused on this topic, myself likely the most confused of all through most of it (though I think I finally “get it”). I had suggested from your previous summary of the ADL thread (which talked about multi-dispatch) that you stopped reading and not fully digested the conversation, but I apologize if you had.
As @tkoolen said, the later turns out not to be true. If ADL works, it is a super-set of the semantics (where all(?) existing code could work). No changes to the subtyping model of dispatch, which would be a dealbreaker. Again, even if it is a strict superset of the existing semantics, that doesn’t mean it feasible to implement. But that is a different explanation for not having ADL that it being fundamentally incompatible with the language.
I think it would be very useful to have a list of those multi-dispatch languages that actually implement namespaces. I went on a hunt a few weeks ago and couldn’t find any precedent for Julia’s approach (although I was told CLOS does?). This may not be surprising because Julia is an unprecedented language on many other levels, and most dynamic languages either have skipped namespaces entirely or have single-dispatch (or emulate operator-overloading with single-dispatch namespace lookups). But this sort of thing can wait.
Many of us have said, again and again, that we want to push off the any revisiting of Julia 2.0 and ADL. Nobody wants to discuss a deign. The reason this keeps coming up are the responses that core developers give on Julia Discourse, which we think makes the issue worse by telling people that this is a conseqnewce of multi-dispatch rather than first-class functions (he real issue).
To roughly paraphrase the current responses from Julia developers on most of the questions on lookups and namesapces:
Julia does namespaces this way because it is a language with multi-dispatch, and you are used to single-dispatch. You will learn to deal with what you perceive as downsides, because this is the only way to get all the wonderful features of multi-dispatch.
As discussed in this thread and the ADL thread, this is false and confusing to people from other languages that don’t see what namespaces have to do with multi-dispatch. The real reason is that Julia chose to do namespaces this way is because it is the natural way to add namespaces to a language with first-class functions.
There is also the following corollary, which comes up in various forms more often than I would like
So learn to live with it, or fork it and get your own users (ed: apologies for the previous explicit)
Luckily this is not from the vast majority of developers, so I will dismiss it.
Yes, on everyone’s list. But a coherent way to communicate the current logic of Julia to those coming from other languages is important now, as is a set of macros to make the transition easier. I, personally, would love to see some simple macros as part of the core standard library to make people be more explicit when trying to merge methods.
Can I propose that a few people work on a PR to try to explain this over the next few months, with the intention that it coudl go into the documentation (in the differences from other languages section)? Then anytime these quetsions come up on Discourse, you can point them to people. I also think there are some natural macros, maybe starting from GitHub - chakravala/ForceImport.jl: Macro that force imports conflicting methods in modules that should be considered as part of the library. Formalizing on this would help everyone to notice the “mild” type piracy inherent in the current design.
The key phrase here is “for a given function”. The entire problem is how to identify which function you’re talking about. For example, if a function is just identified by a name (as it is in class-based OO languages) then the rest of your proposal goes through no problem—we could implement it e.g. by having a single namespace of functions.
Exactly! People coming from other languages tend to think of types and methods having namespaces and functions being a string of identifiers with no intrinsic namespace. This is not how it works in Julia. If it is possible to have functions in a global namespace and methods in the local namespaces is an implementation question that can wait. I don’t know if ADL can work with Julia, and I don’t want to think too much now about it (as long as we consistently communicate to the community as a whole and training to explain the differences).
I, for one, would appreciate a very succinct explanation of what the main issue is and possible resolutions. I’ve been following these discussions off-and-on, and I’ve tried to be open minded as I definitely think there is a real issue here, but to be honest I still can’t help but see it as a very minor one. I come from about 8 years of C++ and FORTRAN (2 years of Julia now) and I have never once gotten really frustrated over any function-namespace-method-extension problem. I think the worst thing that’s ever happened to me is that I declared a function using a name that was already in use, but in every case not only was this easy to resolve but it was obvious how to resolve it, and in many of those cases the name choice was what I’d consider a fairly obvious screw-up on my part anyway. By now I have written a pretty hefty amount of both Julia “package code” and scripts, so it’s getting hard to think that I’m some kind of special case.
It is probably worth mentioning that one place where my view has evolved is that I now understand that there is a spectrum of package types with packages that shouldn’t export anything at one end (e.g. CSV, Feather) and packages that are most useful when they fundamentally alter your namespace on the other (e.g. DifferentialEquations, LightGraphs, JuMP… LinearAlgebra is almost too good an example to count). So, I don’t know, maybe there is some way of formalizing or documenting that concept.
At the risk of sounding too dismissive, I’m starting to think that the rest of this is primarily a documentation issue.
GAP has multiple dispatch and does not really have namespaces. Each function is in the global namespace
and methods are separated by their signature (by default they are protected from overwriting, you must
explicitely unprotect a method with its signature to overwrite it). To have a local namespace, a current practice is to put functions as fields in a record (similar to values in a dict with symbol keys: the
syntax is MyPackage.foo, with meaning similar to MyPackage[:foo]). But then every call must of course be qualified.
I think that is a reasonable perspective. I would say that there are separate things:
Consistent communication and documentation when all these things come up, taking into account the perspective of most users (who do not know about first-class functions)
Macros which may make the current workarounds of merging operators into Base, etc. more explicit and easier to teach
A discussion in a year or two on whether a modified version of the lookup rules could significantly strengthen using namespaces in Julia, make method naming less fragile, make baremodule as a default more usable, and make macro based workarounds unnecessary. Maybe it isn’t even possible…
Not really. Arguably, Python evaluates foo.bar(args...) in much the same way Julia would evaluate the same expression: evaluate foo, then look up the bar attribute of the foo object, evaluate the arguments from left to right, then apply the function foo.bar to the arguments (internally using PyObject_Call). (Of course, in Julia, it’s less common to have the OO style of having objects with function members, except of course for the case where foo is a module.) Functions are first-class in Python, as in Julia — e.g. you can assign a method to a variable via f = foo.bar and later call f(args...). And of course you can have functions in Python that are not OOP methods, and again foo(args...) is evaluated as in Julia: evaluate foo, then args..., then apply. One big difference is that Python has no concept of a generic function object that dispatches on its arguments: given a callable object f, then f(args...) in Python will always go to the same code regardless of args.... The other main difference is that that in a Python object’s method call foo.bar(args...), the foo object is implicitly converted into the first (self) argument in the bar definition—essentially, the foo.bar method object in Python stores a reference to foo—which wouldn’t happen automatically in Julia (though it could be emulated with getproperty).
That being said, I totally agree that a conceptual adjustment is required in coming to Julia from OOP languages or other languages centered around different kinds of abstractions.
I don’t think that is correct, and it is very instructive to see why., which may help understand how to educate people coming into the language
Python looks in the namespace of foo to find the bar method (i.e. an ADL where only the first argument matters), which Julia does not.
So the basic teaching approach is that you tell people coming from python that all they need to do is turn foo.bar(args...) into bar(foo, args...) to use multi-dispatch, which is largely correct (until namespaces kick in).
Lets examine how that works with namespaces.
In python, create a file called mymod.py with the text
class MyClass:
def bar(self, x):
return 1
Then in a separate file or jupyter, go
import mymod
foo = mymod.MyClass()
foo.bar(2)
This works great, and I have been able to use fully namespace qualified creation for my type (mymod.MyClass) But note that there is only one namespace qualification required.
Now, the Julia approach would be to tell people to just convert this by swapping the order. Totally reasonable and easy to teach. So without namespaces it becomes
foo = MyClass()
bar(foo, 2)
But this falls apart with namespaces in Julia. A pythonista told to swap the order with modules has a trickier time
import mymod
foo = mymod. MyClass() #This part works great in julia
bar(foo, 2) #Fails
mymod.bar(foo, 2) #necessary because the namespace of the foo isn't used
This is one of the area of confusion for non-Julia users coming into the language. There are others (e.g. a pythonista might not understand why they can’t create methods of a particular name for a particular type if they use a package).
Here’s my summary of what’s going on in this thread and the other. Take it or leave it. I wanted to write this down so that way people can understand that the responses are not dismissive, but clearly identify the reasons why no one is really gung-ho on implementing this.
Motivation
It starts with some assumption that Julia’s current function namespacing is hard to understand for people coming from single dispatch languages. Example statements:
There’s no objective data on this, just some people saying “yes I found it confusing” (and others not being confused about it, I’m in the latter camp). It’s normally people who are less familiar with Julia dev saying it’s confusing (which is the where the “contentious split” comes from). This then leads to the conclusion by some that further namespacing of functions to allow for extension in new ways is a feature/design that should be worked towards. However, there’s two objections to that.
Objection 1: Utility
First of all, there’s the objection about whether it’s actually better. @jeff.bezanson and I have argued that the resulting code from having multiple functions merge on their own could be more non-local and more confusing. For example:
This is the same thing that I was saying earlier with
So essentially the objection is that the issues you can get from extending functions is really due to abusing existing names and writing something confusing in the first place. I haven’t seen a counter to that in this thread.
Objection 2: Implementation Feasibility
@stevengj, @yuyichao, @mbauman, etc. have engaged in more technical discussions about implementability of these features. While @jlperla brings up here and in Function name conflict: ADL / function merging? that C++ does something similar to what’s proposed, the issue is that there doesn’t seem to be another language that actually achieves this with first-class functions and multiple dispatch. Putting all of those features together is the combination that is difficult to handle. For example:
In Julia, a “function” can be any type with a call added to it. Functions are not distinct from other variables, and names can be dynamically changed since Julia is dynamic. So the implementation has to account for this. So the problem is that @stevengj’s question:
while it’s answered by the simple “it’s easy, C++ does it!” doesn’t necessarily apply to Julia. To keep the nice features of Julia (which are essential to generic programming in this language) and have this kind of merging is completely unknown, and it’s unknown whether it’s feasible. This is why the core devs have proposed technical questions and have asked for PRs / prototypes because it’s unclear how this could even be done. If someone thinks it’s possible, the best way to answer these technical questions is to have a working code that doesn’t have these issues but keeps Julia’s core features in tact.
Summary
So it’s obviously not making it into Julia v1.0. But it’s also not dismissive to say that it’s not a v2.0 priority either. Julia is not MATLAB/Python/R/C++/etc., it’s different in many ways. The way that Julia does generic programming is relatively simple and dynamic which is quite unique. Trying to bring Julia in line with the other languages should not drop this core element of Julia’s design and usage. Objection 1 is basically noting that these changes don’t necessarily help generic programming in Julia. Objection 2 is noting that coming up with an implementation that doesn’t effect Julia’s generic programming structures is really hard if not impossible. The fact that it’s hard to do and doesn’t have a clear positive outcome is why people aren’t throwing their free time on this project.
The Evolution of the Julia Community
I think this needs to be ended with a statement on the current state of the Julia community. To outsiders and newcomers, Julia looks new because you just found it. It’s a language where things are being experimented on, it’s a language which is hot on the “what’s new to learn” lists, etc. People jump in thinking there’s huge wholes in the package ecosystem that need to be filled and it’s the wild west. People call it “in beta” since it’s not v1.0.
However, Julia is definitely not in beta and it’s well into its transition into “not new”. There is an established community. There are large state-of-the-art Julia packages/projects (JuMP, DifferentialEquations.jl, Flux.jl, etc.) which make heavy use of Julia’s unique features. There’s common styles, learning materials, etc. There are people hired to work on and with the language. The language itself is gearing up to release its 1.0 long(er)-term support release in just a few months. And for reference, the language was publicly released more than 5 years ago.
Over these last five years, people have built impressive software using the existing language. It’s popularity routinely lists somewhere around things like Haskell and Lisp. There is no question about whether Julia “has made” or will “stick around”: Julia has already made it and has a large enough community to keep it around, not to mention a stronger funding structure and a larger dev community than any of the other open source languages I’ve seen.
This is where the disconnect seems to happen in these kinds of threads. Newcomers tend to make statements like “it will only be adopted if …”, but everyone around here adopted Julia along time ago and is using Julia not in spite of these differences, but because of these differences. If you think about the issue like that, it’s probably clear why arguments based on “but I know xyz MATLAB/R/Python people won’t adopt Julia until …” is met with “who cares? Julia is for us because we think it’s better, and we’re willing to share it but don’t want to lose what makes Julia unique”. Yes, Julia isn’t as popular as Python, and we can argue endlessly about how to market better (and whether it’s actually a good thing to be that popular at the cost of being so generic, I’m not entirely convinced that’s a good thing). However, arguments about changing Julia will do much better if it’s not discussed like a shiny new toy to play around with, and instead is discussed as the production-ready software that it is.
This has to mentioned because these kinds of threads have a heavy premise in their motivation that for Julia to be “useful” it may need to adopt xyz from some other language to either have people transition or to implement some feature correctly. But this assumption is wrong. People already have and continue to use/pickup Julia because of what it currently does. You can not argue that the Julia should adopt what everyone else is doing without discussing whether it’s better than what Julia is already doing.
Me?
As a specific example for this case, DifferentialEquaitons.jl has worked extremely well because of how Julia’s generic programming works. I am not sure how possible this would’ve been in MATLAB/Python/R/C++/etc., at least without substantial amounts of extra work. Maybe there is a way to work out all of this extra namespacing + auto-merging in a way that doesn’t effect the current generic programming structure at all, who knows. But you aren’t going to convince me that Python does generic algorithms better (it doesn’t), and so if you don’t have a way that keeps Julia’s generic functions over abstract number/array types in tact, I don’t want it. If your proposal does a substantial break to this (and other) existing code then it’s not necessarily a good thing unless I have some very tangible benefits. It’s not dismissive to say “if you want the Python way, then go use Python” because they are different for a reason. I have used both and think Julia’s is much better for building exactly the things I have built.
This is long but hopefully puts everyone on the same page. The technical questions and the community statements are heavily intertwined in the discussion, so I hope this helps parse out why those more established seem to be “against” the idea and why this easily raised tensions.
Much of this I agree with, (though I think your Objection 1 isn’t how it would be, if such a design was possible) and I think you are leaving out all of the the utility of being able to write practical code with any using and without the namespace hacks into Base that may be discouraged in theory, but are extremely common, often necessary, and difficult to explain. The likelihood is that an ADL based approach would make generic programming with namespaces much stronger, but that would remain to be proven. At this point it is not clear if any language with first-class functions has really cracked the problem of coexisting with namespaces in a way that doesn’t require merging all operators into a single namespace.
But all of that is to say that an ADL based approach is something that should be considered down the road. For now, there have been big communication issues because of confusion on where the real problems lie (if they are fixable and the downsides to fixing them could come later). Documentation, education, and macros could help a lot.
Please, come and review / critique / write some code for what I’m trying to do with the @api macro in APITools.jl. I know that even in it’s very early, primitive form,
it’s helped me a lot with the sorts of problems that have been discussed here as I’ve tried to make the Str code as generic as possible, keeping the public API & also developer interface easy to extend.
This is something that I remain extremely unconvinced of. Julia provides both the import and using keywords for a reason. If you are absolutely opposed to using there is nothing forcing you to using it (pun intended). There is nothing stopping you from extending methods from any other package just as you would for Base, as has been said in the past the only thing that makes Base special is that every module has an implicit using Base. I rarely encounter anything I would classify as a “[weird] namespace hack into Base”, certainly not in any of the core ecosystems I’m familiar with.
Sure there is, its name is Julia. We don’t require merging functions, and jamming puns into Base functions really isn’t all that common. Infix operators are indeed tricky, as they aren’t namespace-able, but that’s one of the reasons we have so many unicode infix operators — to allow folks to write compact syntax that doesn’t have difficulties with generic programming.
With ADL, it would simply not be possible to write a cromulent function that uses a function named fit! with our existing ecosystem unless you explicitly enumerate all the argument types that support your meaning of fit!. Ref this comment: Function name conflict: ADL / function merging? - #92 by mbauman
Now, I agree there’s a pain point when two modules export the same (un-merged) name. That’s annoying. That’s why my concrete and simple suggestion above is a tool (editor tool, ideally) that allows you to automatically flush out the names to be explicitly used from a using: …. And there’s a pain point in having a place where package developers can agree on common names to extend, but that discussion needs to happen somewhere for downstream users to make sense of it all.
Extending functions isn’t a hack, it’s easily expressible in the language with its basic constructs. It’s not “discouraged in theory”, it’s directly recommended in the documentation and the associated interfaces. And I don’t think it’s difficult to explain: variables in Julia are in namespaces, and adding dispatches to functions is a pretty core concept of the multiple dispatch structure.
What is not recommended is extending functions on Base types (type-piracy). That’s where Objection 1 comes in. What some people have said is that the problem isn’t language design related, the problem is code design. No matter how you can write it, the fact that you are making some basic name or function act a way that is not how it usually acts will always be confusing. It will always be confusing for a sum function to not actually take a sum (and if it did just take a sum, just extend Base.sum and we have no arguments since any function merging will act exactly the same!). Whether there’s special syntax to allow people to write other sum functions, or whether it’s explicitly discouraged as type-piracy, doesn’t give me a good reason to want a code to ever do this in the first place. If it’s really different, use a different name (or a name in a different namespace), in which case this is a non-issue.
I am thinking about it like whitespace sensitivity in Python. Python requires that you indent your code correctly. There are ways to engineer a language/parser so that’s not required (ex: end like in Julia). But at the end of the day, everyone should indent their code correctly so whichever way you choose is a non-issue or at least a very surface-level issue.
Here we have a coding style choice which isn’t necessarily a clear way of writing code. Even if there is a good way to namespace a function so that way you can have the same name do two things and but have it automatically merge in many cases so that generic programming isn’t broken, you shouldn’t actually use it. If the function is actually doing something different, give it a different name. Or stick a macro on it so it’s clear that this line changes the behavior of the functions. While we have so far had this “accidentally” (?) enforced via how extension works, I don’t see a huge benefit for getting rid of this enforcement. So while I see why this can cause confusion, I respectfully disagree that this is a language problem and not a code problem.
That is only because namespaces right now are used (loosely) to represent concepts, where they are more formally defined in other languages.
There is no reason that there can only be a single active “solve” concept at any point in time operating on a dijoint subset of classes. Being more formal in making sure that these things can never clash is what Concepts/Typeclasses/etc. can do, but that is a Julia v5.0 (if ever). Julia’s namespaces do not do much to help, in practice, with isolating concepts because any time you want to use operators you end up having to agree on a namespace (usually Base).
So you get to use solve in differential equations and the next package that comes around and just wanted to use the word “solve” has to pick a different function? What about operators used in base?
