Possibility of `local import` statements in future?

Internals & Design
1

Is there any possibility that Julia will have local import in the future? I’m aware some people think this is a silly idea; however, this would actually solve a very real problem some users might face.

Namely, people want to extend operations like +,-,*,^, etc on Symbol and Expr types, and it would be desirable to have these definitions in a sub-module of a package to make extensions optional. The problem is, any new module that is defined with import statements extend on a global scope before using it.

Therefore, it would have language applicability if the import statement could be handled locally, and then optionally extended to the main scope with using. I’ve been told it isn’t possible with Julia, but I am optimistic. It would require some kind of a table that keeps track of which method extensions are available in what scope. Saying that it is entirely impossible seems rather improbable to me.

Perhaps, having this could speed up the method-look-up in some cases, since there would be fewer extensions to search through in a given scope.

Before you try to shove this issue under the rug, consider that multiple packages that all define arithmetic operations on symbols and expressions cannot be loaded at once because of this issue. However, if local import where possibe, then different packages that depend on different extensions of the same methods can be used together without interference (because then scope determines the method extension selection).

2

Just use assignment, like + = Module.:+.

Method lookup does not care about any names.

3

That replaces the + function, doesn’t extend it.

4

Essentially, what I am noticing is a greater extent of context sensitive ambiguity in the Julia language. What prevents the Julia language from truly being able to have a 1-1 correspondence with mathematical vernacular is its ability to discern context sensitive ambiguities. Mathematicians are able to do this, but most programming languages, including Julia, cannot.

One example of contex sensitive ambiguity is the fact that Julia cannot discern between the overloaded definitions of + by scope.

Another example is that Julia is unable to discern the context sensitive ambiguity of order of operations for the definitions of + by scope.

If you wanted to be able to turn Julia into an extremely flexible theorem-prover type of mathematical vernacular / language, you would definitely have to be able to discern those two examples of ambiguity.

The one common trait of all these ambiguities is that they are contex-sensitive, i.e. they should depend on the scope of the evaluation, i.e., determined by what is in the local namespace.

Mathematicians, for example, make many new definitions of the same symbols in different contexts, and in these different contexts you might need different definitions of the same exact thing, further with possibly diferent order algebraic of operations or associativity.

These are some things that the language designers might want to think about, in my hope…

5

It’s the same as global import. You could still extend the + with eval (method definition must be at global scope.)

6

For example, what if I want the + to work on both numbers in the regular way, and I want the + to also be defined for Symbol types, but only in module Test2. If I am outside Test2 or have not imported it, I do not want the definition of + that also takes symbol arguments, but I still want the standard +.

module Test
  module Test2
    Base.:+(::Symbol,::Symbol) = 2
  end
end

:x + :x # 2

It doesn’t work, there is currently no way to do it. Without using Test or using Test.Test2, the definition for + has been updated at a global scope. Now in all scopes it gives that result. But what if in another package somebody wants to extend the Base.:+ function on Symbol types differently. Then you cannot have both extended + functions simultaneously, even though they are in different scopes. Theoretically, one might want to be able to have both of those simultaneously, but Julia would have to discern the scope.

7

As I said above, you’ve mixed up name lookup and method dispatch. The two are completely unrelated. If you want to have two things (functions) to dispatch differently, they cannot be the same object. Therefore, what you need is a different +, which you can get by +(::Symbol, ::Symbol) = 2; +(args...) = Base.:+(args...). You can then use the + you’ve defined that’s independent of Base.:+ in whichever scope you want.

1 Like
8

What about the following:

module Foo
    +(a,b) = Base.:+(a,b)
    +(a::Symbol, b::Symbol) = 2
    println( 1 + 2 )
    println( :silly + :bar)
end

Doesn’t that work as you’d like?
You have a version of + that is specific to module Foo, (you can even use it outside of Foo by writing Foo.:+)

6 Likes
9

The problem with that is that you do not retain the 180 methods of multiple dispatch from +. Your version only has the Any defintion of + and the Symbol definition. I suppose that could work though.

The point of it is to have an extended version of +, but with possibly different extensions in different scopes.

With your solution, the user has to manually call + = Foo.:+ for every single extended function to import in the new scope, when using Foo. But the user should be able to do something like using Foo or importall Foo and have the new definition of + instead of the Base definition. But that doesn’t happen, making it impractical when a large number of functions needs to be treated this way.

10

Yes you do. Did you try what yuyichao wrote?

11

Yea, tried it. Like I said, I suppose it could do, but it is not properly extending the function and it cannot be imported en masse for a large number of functions without having to evaluate many more assignments.

module Foo
    export +
    +(a,b) = Base.:+(a,b)
    +(a::Symbol, b::Symbol) = 2
end

julia> importall Foo
WARNING: ignoring conflicting import of Foo.+ into Main

Instead, the user has to be asked to do + = Foo.:+ for every single function. Theoretically, one could create a function that generates the code to do all those assignments for you, but the user would still have to enter something like eval(Foo.generated_assignments()), and evaluated in the Main scope.

julia> module Foo
           export +
           +(a,b) = Base.:+(a,b)
           +(a::Symbol, b::Symbol) = 2
           generated_assignments() = :(+ = Foo.:+)
       end
Foo

julia> :x+:y
ERROR: MethodError: no method matching +(::Symbol, ::Symbol)

julia> eval(Foo.generated_assignments())
WARNING: imported binding for + overwritten in module Main
+ (generic function with 2 methods)

julia> :x+:y
2

It would be preferable if the language itself could handle this properly.

Specifically, instead of ignoring conflicting imports, it could replace them instead. Then this would change how everything works slightly, but would be more flexible perhaps. Or a new keyword should exist so that the imports can be forced to overwrite the existing function, like local import Foo, but with the same effect as having eval(Foo.generated_assignments()). That should’nt be too difficult to add to Julia, right?

People said it was impossible, but here we are. Looks like a local import can happen, it only needs to be part of the language now.

12

What do you mean by “properly”? I’ve already proved above that you cannot extend Base.:+. What you want is fundamantally different.

You can do import Foo: +. Also, how is that different from import? You defining conflicting names and it’s far more clear to be have to be explicit about it. For anything that’s not base, using and importall are extremely confusing anyway.

And if you really don’t want to be explict, just use a macro is much cleaner.

There’s really nothing local about it though. It’s just implicit about what you’ve imported.

13

Sure, you can do that. But then the user has to explicitly list possibly 50 different function names if he wants to import all of them.

Actually, when you want to import a lot of functions, it would make sense to be able to import them all simultaneously to overwrite all of them at once. Whether it confuses you or not does not matter, since some people will actually need to use this, and not having it part of the language and having some kind of hack with eval(Foo.generated_assignments()) would be even more confusing and less obvious than a simple import statement that imports all of them and replaces everyting “locally” in whatever module it is imported.

The point is not about it being local, the point is that Julia doesn’t have a feature that does that. And by the looks of it, it would be a fairly simple feature to create.

14

Also, just to be clear. I never said it was impossible. In fact, I’ve showed you in the very first reply that local import (import a name to a local instead of global scope) is possible. What I did say was impossible was locally extend, i.e. extending (mutating) an object but has that effect only visible in some scope for the same object. But I then show that it’s just defining new functions and you can do that just fine. The only problem is that you want to do something fundamentally confusing (ambiguious?) so you have to make a choice.

Which is usually a good thing. Especially in this case. It’ll be extremely hard for the user to figure out which functions (and therefore which sets of methods) are actually being used in a scope. Copy pasting the import statement is literally the simplest thing to do and the easiest document for what is being used.

This is a hack only because you didn’t use a macro. The only thing that’s confusing for the reader is that your + is different from everyone else’s without an obvious clue where that comes from. As long as you decide to hide the actual symbol, there’s nothing around it. It doesn’t matter if it’s a builtin feature or a user defined one.

And the point is that we don’t need that feature (edit: I mean we don’t need to implement this as a builtin feature since it can be done by the user easily and doesn’t offer anything by being builtin as shown below. I did not mean that the features and implementations I’ve shown above are features of the language to be removed.). It doesn’t really help to clearify anything compare to existing implementations. You can even make this more generic and implement a force import macro yourself if you don’t care about readers that haven’t used your library before (If you are using the library all the time it won’t be confusing at all but in my experience, anything like this will become pretty confusing if you stop using this for a few weeks/months). You can just spell it as @force using Foo and you can check how it can be implemented by looking at Reexport.

15

On github you explicitly called me out and said it was impossible some months ago. There is a record of it online of you saying it is impossible, but whatever, deny it if you want. I’m not interested in making any personal comments, and I did not say in my post above that it was you, but now we have brought you into it

This would be explained in the readme and the documentation. I dont recommend using a package before reading the readme and the documentation. Of course, it will be made clear what exactly is happening.

The precise purpose of this is so that thse functions don’t get automatically imported. By default, they should be skipped. But a user should be able to do force importall Foo and get all of them if they really want to fully extend all the functions provided. There are literally around 50 or so functions that this needs to be applied to, so it is very impractical to ask the user to explicitly import 50 function names each time the module is loaded into the REPL, for example.

This would seem like a pretty standard feature to have in a language like Julia.

16

Judging from that the initial use of word was not accurate in this thread, I won’t be surprised that without clearification I’m talking about something different from what I understand you want to do in this thread. I’ve only found some possibly related discussion in Rename parse(::String) · Issue #24349 · JuliaLang/julia · GitHub though the whole point there was to not make them the same function rather than having them as different function to satisfy some other requirements.

What’s the difference between that and @force using Foo or @force importall Foo?

Again, the point is that if forcing name overwrite is the only thing you want, having it in a package is perfectly fine and usually preferred.

17

How can Foo provide the @force macro before using Foo is executed?

18

@force can be provided by a small package (just like Reexport) to provide this function for all other packages.

19

That could be one way to do it, but why can’t the Julia language just provide that macro natively?

Then it can be placed into Compat for older versions.

20

Like many other features, if a feature can be provided by a package, it should. If it is later proved that the feature is of significant interest (which I kind of doubt this is of that general interest even though I don’t deny that it may be of interest sometimes) it can become a standard package or be moved in base.

4 Likes