Possibility of `local import` statements in future?

[yuyichao]

@TsurHerman

I am not in it for admiration, but for open mindedness ,learning ,the joy of solving abstract stuff together.

And you disappoint me because you are obviously capable and probably better at this stuff (with respect to Julia internals) than me, and yet you too easily regard any idea that is different than the status quo as being wrong.

Feel free to advice how I can point out all the issues I’ve seen without disappointing you. FWIW, if I’m not open minded to what you are proposing, I would have just straight out calling it non-sense, useless and ignoring it. I do agree it has some properties that you’ve claimed and I don’t believe that need any further discussion. (Discussiong things that are clear enough is what I meant by admiring above, especially when it’s about the positive aspect.) The only thing that need discussion is what you didn’t mention, including the support for all current usage and the real impact on the compiler. Those are exactly what I was tallking about. The fact that I didn’t make any positive comment on that is simply because from the basic outline you’ve given I can only see what problems they can cause without an obvious way around it. I would certainly have mentioned possible way out if I come up with one. If there is indeed one, then I still believe explicitly showing the problem is the best way to promote you, the only one who really know what you are actually proposing, to come up with potential methods to solve them.

@TsurHerman

in this example the start(iter) method called from forloop which is called from Module Main will resolve correctly.

OK, so, still the same issues. It works only because the caller is actually a module that is aware of Foos, which have two immediate problem that I’ve pointed out above,

1. There’s a version of forloop that’s aware of Foos and a version of forloop that’s unaware of Foos so that the code will run correctly when calling from different context. That’s the compiler problem that I’ve mentioned repeatly above.

2. It’s still impossible (AFAICT) to move part of the implementation of forloop to a different function or different module. As a concrete example, if forloop is defined as,

   forloop2(...) = <your current implemenntation>
   forloop(...) = forloop2(...)
   

   and only forloop is exported. Then according to your rule, forloop2 will only resolve function in ForLoop and not Foos (since only forloop will be called externally and forloop2 is only called from forloop, which is in ForLoop). Note that this is exactly the f and f2 I’ve used twice above.

   This is made even worse if you want to extend the function of forloop by wrapping it in another function forloop3 in another module ForLoop3. You can improve these better by chaining all the way up but 1) that’ll be a pretty confusing rule IMHO (since it’s extremely context dependent, think dynamic scope) and kind of back to where we are in most cases if you start from Main. Depending on the exact rule, it’ll also make compiler problem even harder since you’ve got a whole chain of context to consider before compilation…

   Do note that all of the usecases I’ve mentioned above (forloop2: implementation detail helper functions; and forloop3: wrapper functions) are existing and real usecases that must be supported.

I’ve often found that @yuyichao is not open to new ideas that are different from what’s currently the status quo

I do agree with this to some degree and while I have proposed and implemented big changes before, I hold all my proposals to the same standard that it need to withstand every current usecases that I can come up with. For all usecases that should be supported, it should be either unaffected or can be achieved relatively easily in a slightly different way.

Note that this is different from @force. It’s not a feature with huge impact. It’s just the base policy to not put every small things in as long as it doesn’t have to be. AFAICT, putting it into base/add as language feature is the only thing I’m against about it.

[TsurHerman]

@yuyichao

forloop2(…) = <your current implemenntation>
forloop(…) = forloop2(…)

if forloop2 is something that needs to be extended (like IndexingStyle etc) then yes it needs to be exported.
Base is pretty flat structure , I think you’ll find a lot of “part of the implementation” functions exported anyway

And yes it is context dependent not just on the types (multiple dispatch) but also on the calling context.
So you can have separation of types from interface implementation.

You can have one module defining graph types and two modules defining the implementation of the iterator interface differently and you could have:

module A
using Graphs
using GraphIterator1

#do some things involving iterations over graphs
end

module B
using Graphs
using GraphIterator2

#do some things involving iterations over graphs ..differently
end

Regarding the binary code and compilation, as I mentioned in one of the earlier comments … part of the reason
to make the effort , is that this method guarantees that for a method f in module M and arguments args the binary code will not change as long as you don’t change (f ,M , args) , caching is clear … no type piracy.

there can be true incremental precompilation in the sense that if module A and module B both use StaticArrays then the binary/IR code of a functions in A and B involving StaticArrays … belong to the cache of A and B and not StaticArrays.

[yuyichao]

@TsurHerman

if forloop2 is something that needs to be extended (like IndexingStyle etc) then yes it needs to be exported.

Base is pretty flat structure , I think you’ll find a lot of “part of the implementation” functions exported anyway

I assume by “needs to be extended” you don’t me extending the function in julia sense but seeing user definitions. In that case “part of the implementation” functions are basically every functions and they are clearly not all exported. As an example, just look for all methods starts with _collect here (there are two functions and many methods). They all need to see the user defined iterator protocol.
Or really, just look for any function in base, they (edit: almost) all need to see user defined methods.

Also, according to what I understand your proposal to be, exporting the function doesn’t really affect what the function see when called from the same module. Even if forloop2 is exported, as long as it is called from forloop and not by the user directly, it still can’t see the scope of the caller of forloop.
That is, unless by caller you don’t mean caller but the toplevel scope (current_module() on 0.6) or the first caller from a different module or something in between. Each of them has their version of the same problem and if you indeed me one of those, you should clarify so that I can be more specific. Listing them all here will be too long… (edit: though I just realized that I inevitably listed most of them below…)

@TsurHerman

You can have one module defining graph types and two modules defining the implementation of the iterator interface differently and you could have:

OK, so as a more concrete example of what I mean by the problem when you have multiple modules involved in a way similar to what you’ve listed here.

# All modules are using all other modules that the module is aware of so no new using/import can be added.
module IfaceDefAndUse
# This module cannot be aware of any other modules
export g, f
function g end
f(a, b) = g(a, b)
end

module WrapperOfIfaceUse
using IfaceDefAndUse
export k
k(a, b) = IfaceDefAndUse.f(a, b)
end

module IfaceImpl
using IfaceDefAndUse
export T, f
struct T end
IfaceDefAndUse.f(::T, ::T) = 1
end

module Library
using IfaceDefAndUse
using WrapperOfIfaceUse
using IfaceImpl

export l()
l() = WrapperOfIfaceUse.k(IfaceImpl.T(), IfaceImpl.T())
end

module User
using IfaceDefAndUse
using WrapperOfIfaceUse
using IfaceImpl
using Library
Library.l()
end

Now, how can IfaceDefAndUse.f(a, b) know which g to call? The direct caller of IfaceDefAndUse.f is WrapperOfIfaceUse.k, which isn’t directly aware of IfaceImpl. It’s caller, Library is, but there isn’t any reason to pick this module to use unless you want to include all modules in the call chain. User as it is now, is also aware of IfaceImpl and it is the toplevel caller so there’s some reason to treat it in a special way, but there’s no reason User need to using IfaceImpl (in practice, there’s a lot of reason you don’t want to using a dependency used by a module you use) and if you don’t, you can’t pick it up from here either.

So for this usecase, I can see this working only if you make IfaceDefAndUse aware of all modules in the call chain, or making it aware of the toplevel module and require the toplevel module to import all modules. Either way though, it’ll require the function to be aware of a lot of other modules/context in this particular call chain and this is exactly what I mean by making the compiler problem worse.

@TsurHerman

s that this method guarantees that for a method f in module M and arguments args the binary code will not change as long as you don’t change (f ,M , args) , caching is clear … no type piracy.

Except that M is a long list of modules making the compilation result not reusable at all. Type piracy is also not really a compiler problem, it’s a code organization problem. Compiler is totally fine with it. Only the user will be surprised.

@TsurHerman

there can be true incremental precompilation in the sense that if module A and module B both use StaticArrays then the binary/IR code of a functions in A and B involving StaticArrays … belong to the cache of A and B and not StaticArrays.

This is really solving a non-problem. Being able to incrementally precompile isn’t even a problem. The usefulness of the result is. You can always do compilation and nothing can stop you from doing that. The only question is how useful the compilation result is for future use (i.e. the reusability of it). Doing compilation without maximumly reusing the result will cause strictly worse performance.
What you are suggesting here is basically that by making things more context dependent there are a lot more to compile and by construct those compilation are more independent (there’ll be much less sharing). That’s exactly how you make it less compiler friendly and kill the performance.