Code loading and namespacing are not necessarily the same thing. They are two distinct features of a programming language. It’s one thing to say “now run this code”, and another to say “this code is namespaced, and now I want to reference these pieces from the name space”.
The underlying issue here is that some languages put these together and make their code loading syntax add namespacing. They are two different pieces, but the syntax only lets you tell the interpreter do both together. That is a nice convenience for a lot of things, so it’s done that way for good reasons. Maybe that “loading with namespacing” it should be done throughout more Julia code. Though I do believe that keeping access to functionality that is specifically for code loading does have its own purposes. And that I think is the tension in a nutshell.
I’m confused as to how that resolves @jzr’s comment though? Yes code loading and namespacing need not be the same thing, and being able to inject a bunch of code into the current namespace with include is pretty useful. The sticking point is that the code being injected also automatically gets access to everything in the parent scope. It’s the worst kind of dynamic typingscoping IMO, because it’s context-sensitive and can’t be trivially analyzed. Any file can be included 0…inf times in 0…inf places. Provenance becomes much harder to determine, and (as I mentioned on the GH issue) users are forced to read from the top down or linearly back to beginning in the include order just to find where a given function or constant is declared.
What I’m saying is that is a necessary side effect because include is simply Julia’s core code loading procedure, and that has a very distinct purpose. There is no place to reference a name from whence thing came because there is no “whence it came”: it is just the thing defined in the command that comes next. It’s exactly the same as pasting into the REPL. Whatever can do such namespace adding is separate from the primitive for inducing code loading. And maybe that code loading primitive should be thought of as an internal compiler detail and maybe users should be instructed to mostly interact through a separate high level interface, but that does not obviate the need for include.
Everybody keeps on talking about the dangers of including a file multiple times, but I don’t understand why this would even happen. Is it really that hard to just put all of your includes in the top-level ThisIsAPackage.jl file (and nowhere else) and then be done with it?
I’ve read the #4600 issue, but even now it seems to me we are looking for a problem where there really is none? Can anyone provide an actual example of such a problem caused by the current use of include/using?
Edit: I mean, anyone can construct some weird case by defeating common sense, but what about a real-world example?
This would be the ideal scenario! Essentially state caveat emptor when working with include and get folks on board with using whatever higher level interface materializes.
I don’t know how to make the following sound not facetious except by prefacing that I mean this as a totally good faith genuine question: if having a resolver on a [directed acyclic] graph is spooky implicitness and a easy source of bugs, should we get rid of the Pkg resolver and make users explicitly declare every version of every direct or indirect dependency for their projects?
Presumably not; what are the relevant differences between the Pkg resolver and this case?
It is a source of a lot of issues for exactly this reason. But its convenience far outweighs the other option of doing it by hand. I think the calculus in Pkg’s case is a lot more slanted: working out package dependency versions by hand would be an absolute nightmare. Or the other alternative, always use the latest released version, is a nightmare of non-reproducibility. So I would think in Pkg’s case, it is well-worth the trade-off, though a trade-off is made and sometimes it’s frustrating.
I suspect this is mostly because of the convoluted way imports are resolved to actual files to load? I actually quite like the simplicity of one module equaling one file otherwise, together with relative imports.
I would already be happy if https://github.com/julia-vscode/julia-vscode/issues/307 would be fixed.
The fact that this is an open bug for more than 3 years is for me an indication that the Julia language is missing the features to implement it.
What I am missing from these proposals and discussions is the recognition that the status quo works for a large number of Julia users.
I recognize that people have different coding styles and habits from other languages, but the idea that the current implementation of modules is somehow fundamentally broken (as suggested eg by the title of this topic) is unlikely to lead to productive discussion.
I would suggest that if someone wants to propose an improvement, they should provide a concrete example that can be discussed in detail. Not necessarily a toy example, but eg link to a project that is somehow a mess because of how modules / namespaces work in Julia, then entertain suggestions on how to organize it better with the language and the tooling as is.
I certainly agree that concrete suggestions lead to better discourse. Regarding the last point of finding a project that is currently a mess, it’s quite possible that a given system’s problems are in effort required, not eventual outcome. (FWIW, I am agnostic on whether the include system falls into this category.) Or, more concretely, perhaps we all spend more brain cycles than ideal thinking about the order of include statements and related ideas, and some other system would produce equally robust results with less effort.
I would go so far as to suggest to the OP editing the title to something less aggressive, if this is going to be the thread to discuss future improvements. Suggesting that something is not “proper” is the ultimate criticism, if I judge by how my grandmother uses the term.
There is only one way to organize linearly, but an infinite number of ways to organize non-linearly: and of course non-linearity quickly leads to chaos. Don’t the packages already mentioned resolve the main issues?
encapsulates code and data to implement a particular functionality.
has an interface that lets clients to access its functionality in an uniform manner.
is easily pluggable with another module that expects its interface.
is usually packaged in a single unit so that it can be easily deployed.
Note that there is no mention of dependency resolution or code loading. As you can see, there is nothing improper about Julia modules, so the title of this thread is definitely misleading.
The focus of this discussion is on dependency resolution/code loading, which is somewhat orthogonal to the use of modules as namespaces.
I’ve argued before that using submodules in Julia is an anit-pattern, because what one should be doing is overloading generic functions. I would much rather have
foo(x::A)
foo(x::B)
than have
A.foo(x)
B.foo(x)
As @jonathan-laurent admitted in his Github post, using a lot of sub-modules adds a lot of boiler plate to a package, since you’re constantly importing and exporting types and functions from various submodules within your package.
Some folks have argued for the file-module correspondence (each file is automatically a module). I strongly oppose this. Since I don’t use submodules in my Julia packages, I would be forced to put the entire codebase for a package into a single file.
If future Julia code ends up looking like the following, then I’m out.
I believe that this debate reveals a cultural gap between people coming from a software engineering background and people whose culture is more solidly rooted in scientific computing and scripting. The Julia community needs both kinds of people to thrive.
It should not be controversial by this point that the current module system has a number of properties that can make it relatively harder for people to understand and maintain large Julia codebases, while also failing to incentivize good and established software engineering practices for package developers. At least a dozen of people have made this argument eloquently in various threads and I also added my two cents to this debate by explaining how these issues concretely affected my AlphaZero.jl package among other things.
To be clear, no one is asking for include to be removed from the language or questioning the workflow of the many people that are extremely productive using Julia as it currently stands.
The one thing that Julia teaches me is that not everything has to be a tradeoff. In the same way that Julia showed the world that the same language can deliver on both performances and ease of prototyping, I do not see any fundamental reason that it should not be able to evolve in a way that makes it more natural for package developers to adopt better software engineering practices without sacrificing anyone’s workflow.
I, for one, find the concreteproposals that are being made in this direction very interesting and my bet is that this will result in a system that is overall simpler, works better for everyone and even enables better tools.
Just because a particular software engineering practice is “good and established” for some other languages does not mean that it is automatically the best practice for Julia software. Proponents of functional programming like to claim that functional programming obviates the need for all or most of the OOP design patterns advocated by the Gang of Four. In a similar vein, perhaps multiple dispatch (along with first-class functions and data types that are not bundled with methods) obviates the need for deeply nested modules a la Java and Python?
That’s the exact opposite of the impression that I’ve gotten from this discussion. Sure, include could be left in the language, but it seems that what some people are advocating for (including yourself) is dependency resolution and code loading that happens without using include (and for this to be the new normal way of doing things).