and now if you include them all in MyPackage.jl? or C/C.jl includes D.jl and MyPackage.jl includes C.jl? If you are doing the later then you are now only one step away from @StefanKarpinski 's proposal in issue 4600. ( the early proposal use this convention and let using work on it automatically)
The problem arise when the number of files grows in a project, everything is easy when you only have a small number of files. And as @mbauman says, people who advocate this mainly want things to be resolved automatically.
But how hard is it to just support it? There are already many proposals and implementation attempts. If one doesn’t like implicit modules, we can just not do it, but we can still resolve the file dependencies automatically, I just don’t get it why people want to do this by hand? is this fun or something?
I mean even with this supported, one can still use include if preferred to exercise their brain. So why you guys against this change so much?
This is not a very effective line of argument and is generally unconvincing when advocating for additional language features. Every additional way of doing a given thing adds complexity to the language — complexity that frequently makes it harder to learn, code, support, and maintain.
It’s far more convincing to acknowledge that cost and instead demonstrate how it makes certain patterns easier and more straightforward.
Yeah, that’s fair, but I think removing some brain burden is an advantage isn’t it? If we go with Stefan’s proposal this doesn’t even create new syntax. And I think some people do aware that include is annoying based on your JuliaHub search results right?
Or on the other hand, consider this is a syntax sugar of writing the code manually above. (if we decide to go with that convention)
What’s the difference between C.jl and C/C.jl?
If that’s how you structure your modules/packages, then I understand why keeping everything in order is difficult
But this is exactly the example of the convention @StefanKarpinski proposed. I just copied from issue 4600. And this can happen in the real world, where YaoCompiler contains two IRs with their own intrinsic, I’d like to keep the intrinsics in a module Intrinsics so this results in something like YaoCompiler.HIR.Intrinsics and YaoCompiler.Intrinsics, I later moved them out of the mono repo, but I think this can happen if two people writing the same project on different module, they can have submodule that has the same name, this is reasonable.
Or of course we can reject this convention and file ordering, but it should error explicitly
And when one considers a general solution to the language, one should consider these corner cases to conduct a good solution.
On the other hand, human-managed file dependencies can have the well-known human sanity errors, since every human can make mistakes, while machine can guarantee on things.
edit: @doit oh sorry I was wrong, it should be C/C.jl conflicting. I should note that this shouldn’t work, but rejects. What I intended to demonstrate is the file ordering
where D is a sub project, so in principle one would want to include("C.jl") in D/D.jl this is what has been proposed in the issue 4600. If we are doing your convention proposed above, this is exactly what was proposed to be handle automatically in the early proposal there.
My point still stands, what is the difference between C.jl and D/C.jl?
If they are functionally the same, keep only one. Then, you can also apply the methods already shown.
(I would like to get the point, I just don’t see a circumstance where you would actually do something like that.)
As my example above, they can be completely different things - different intrinsic for different IRs for example (hope this example isn’t too technical), when your IR type definition are in different module, putting an Intrinsics inside each reads nicely like XXXIR.Intrinsics and it helps one distinguish whose intrinsics is that.
Why do they then not have different names? Or the other way around, why not join the code? (As far as I see, if the name is the same, they somehow do similar things but maybe to different objects?)
you need different module to distinguish whose intrinsics is that. I can just check on the module identity using === to distinguish them, without conflicting, while it still reads nicely and whoever wants to import all the intrinsics only need to remember to call using XIR.Intrinsics
That’s not possible - at least in my case, the intrinsics are different functions handled specially at compile time, only their module, type signature is known, putting an extra context type create a lot more burden - I have to look into the methods and specialize on the types, or maintain a list of intrinsics instead of just do === on its parentmodule
As I’m aware of this problem, I’ve splitted them into two packages and try to use a different name, but again, this experience wasn’t very pleasant.
I can understand your frustration with that. Without a look at your code and then also taking the time to understand everything properly, there is no way of telling (for anyone else) if another way would have been possible or not. (But that is kind of off-topic …)
If you’d like to continue this, it’s probably past time to heed that warning about continued replies to the same person and discuss this directly between yourselves, @doit and @Roger-luo — either via personal messages here or on a more direct chat platform like the Julia Slack or Zulip.
As a passive reader of this exchange who is eager to understand the crux of the matter, I would like to commend everyone involved in the recent discussions (last day or so) for being cordial and patient. It’s becoming clear that different people have very different workflows and mental maps of how to organize their work, and hence different needs. Based on recent exchanges, I’m beginning to see one of the main issues as solving the “include-twice problem”. And if my understanding is correct, one of the main causes of this problem would be for collaborative projects. Certainly we can agree that including twice is a real world thing: the question is how to resolve it: what could be the downsides to automating the resolution? To use an analogy: resolving packages with conflicting compatibility requirements is very hard to do automagically and often best done by human intervention…
Isn’t that intrinsically impossible to includes be sorted automatically? I mean, we can come with many different examples, but if these commands were in different included files of course the result is not the same, no automatic algorithm could figure out what the programmer wanted:
julia> f(x) = 2
f (generic function with 1 method)
julia> f(1)
2
julia> f(x::Int) = 3
f (generic function with 2 methods)
julia> f(1)
3
julia> ### start again
julia> f(x::Int) = 3
f (generic function with 2 methods)
julia> f(1)
3
julia> f(x) = 2
f (generic function with 2 methods)
julia> f(1)
3
I hope someone capable thinks with care about the sub-package possibility, with the “Main” package managing its own local registry. That will be very powerful for code reuse, have all the advantages, and non-breaking.
It’s not anywhere close to being as complicated as a local registry. Or even needing to “resolve” dependencies. It’s just about including files once and having a short idiom to do so — and possibly wrapping the included code in a module. There’s still a lot of design work to be done here in terms of how the new feature would actually behave, though, so it’s hard to say much more than that.
I’m just grateful that on a scale from 1 to python this request seems to be in the fairly low numbers.
Let's not do this:
[Olympus:~/Projects/😱py]
$ ~/.julia/conda/3/bin/ipython
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import numpy as np
In [2]: np
Out[2]: <module 'numpy' from '/Users/mbauman/.julia/conda/3/lib/python3.7/site-packages/numpy/__init__.py'>
In [3]:
Do you really want to exit ([y]/n)? y
[Olympus:~/Projects/😱py]
$ echo "print('oops')" > numpy.py
[Olympus:~/Projects/😱py]
$ ~/.julia/conda/3/bin/ipython
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import numpy as np
oops
In [2]: np
Out[2]: <module 'numpy' from '/Users/mbauman/Projects/😱py/numpy.py'>
I am not sure about this (my impression is that submodules are not used that widely), but indeed it is a perfectly reasonable approach to organize a package if someone wants to do it.
What is still not clear to me from all of these discussions is why a bunch of linear includes are not adequate for this task. Of course this involves the coder manually figuring out the order, but I am not sure it is that much of an inconvenience (at least people who maintain large packages, eg @PetrKryslUCSD, say that it isn’t). Or from a different perspective, if a package becomes so large that this doing this manually is no longer manageable, then this may be the least of its problems.
Thanks for collecting these — having concrete examples would make this discussion much more focused.
However, the first two just document that including files has to be in right order, which applies generally in Julia regardless of submodules. Eg in
module Foo
bar(::SomeType) = 1
struct SomeType end
end
one gets a ERROR: LoadError: UndefVarError: SomeType not defined. Order can matter in Julia code.
The third example is about something similar, about docstring of a method in ImageFiltering.Kernel referring to a name in ImageFiltering that happens to be defined later. But I am not sure how that would be related to this discussion: presumably a submodule A.B is within A, so it could not load A “first”.
Again, while you are undoubtedly right that
it would be nice to see concrete examples of idiomatic and mature Julia code where the internal dependency structure is so complex that automating inclusion order is really needed. Given that even complex modules like Base work just fine with include, stronger arguments in favor of this change would be informative.
This seems a little unfair. First, a feature like this is never really “needed” — it’s just a convenience. (You don’t “need” namespaces at all, for that matter—just prepend MyModule_ to all of your symbols.) Second, it’s a bit of a chicken-and-egg problem — because we don’t currently have automated inclusion of submodules, that actively discourages complex internal dependency structures in idiomatic Julia code.
People already use complex dependency graphs for modules, but they currently accomplish this in Julia mainly through modules segregated into packages. It seems reasonable to make this easier to also do internally within a single package.
Supporting (e.g.) using "Foo.jl" would be a non-breaking change, would simplify a common pattern of include("Foo.jl"); using .Foo, and would allow package-like complex dependency graphs for internal modules, which seems like a very natural feature for some people to want.
It’s becoming clear that different people have very different workflows and mental maps of how to organize their work, and hence different needs. Based on recent exchanges, I’m beginning to see one of the main issues as solving the “include-twice problem”. And if my understanding is correct, one of the main causes of this problem would be for collaborative projects. Certainly we can agree that including twice is a real world thing: the question is how to resolve it: what could be the downsides to automating the resolution? To use an analogy: resolving packages with conflicting compatibility requirements is very hard to do automagically and often best done by human intervention… 