# Implicitly loaded modules in the future?
**URL:** https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646
**Category:** Internals & Design
**Tags:** question, module, code-organization
**Created:** [June 9, 2021, 8:58pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646 "2021-06-09T20:58:00Z")
**Posts on this page:** 20
**Page:** 6
### Author:  [@jonathan-laurent](https://discourse.julialang.org/u/jonathan-laurent)
#### Post date: [June 12, 2021, 4:44pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/101 "2021-06-12T16:44:43Z")
I completely agree with you that this debate has been conflating many important questions about code loading, dependency resolution and module systems. These questions are not completely orthogonal though as the current code loading mechanism dictates a lot of what people can and cannot do with modules for example.
> [@CameronBieganek](#):
>
> I’ve argued before that using submodules in Julia is an anit-pattern, because what one should be doing is overloading generic functions. I would much rather have
This point you are making that multiple dispatch works better with flat namespaces is very interesting and it gets me seriously thinking. I am not fully convinced that a conflict between multiple dispatch and submodules is inevitable but this should definitely be a topic for further debate.
Still, it remains that submodules are used by people because they fulfill a need and we should not be satisfied with dismissing them as an antipattern without offering alternatives. Concretely, I would be genuinely curious about better ways to organize AlphaZero.jl. I cannot see any better way under the current status quo and this is one reason some of the [proposals](https://github.com/JuliaLang/julia/issues/4600) that are being made are appealing to me.
> [@CameronBieganek](#):
>
> If future Julia code ends up looking like the following, then I’m out.
Fortunately, there are many ways to enforce clearer internal dependencies without falling into the excesses of Java. I can promise you that I am as invested as you are into not turning Julia into a Java clone. 🙂
---
### Author:  [@jonathan-laurent](https://discourse.julialang.org/u/jonathan-laurent)
#### Post date: [June 12, 2021, 4:46pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/102 "2021-06-12T16:46:51Z")
> It sounds like you’re implying that the people advocating for the status quo only write research code…
This is an error on my side. Please allow me to edit my original message.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [June 12, 2021, 5:01pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/103 "2021-06-12T17:01:12Z")
> [@jonathan-laurent](#):
>
> Concretely, I would be genuinely curious about better ways to organize AlphaZero.jl.
This software is amazing! But since you asked: I don’t think you are using modules to their full advantage. For instance I found that having to explicitly state which quantities are being used in a module is invaluable in making the code in the module self contained. That is helpful in making the code more legible and more easily understandable. When you have files without modules which are included who knows where, the functionality used in that file is wide open to outside influences. Hence, the code inside the file is harder to understand.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [June 12, 2021, 5:08pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/104 "2021-06-12T17:08:19Z")
I have no idea how feasible this is, but it would be cool if a future version of Julia allowed referring to types that haven’t been defined yet. It would be especially cool if this allowed defining mutually recursive types.
If we had this feature, then all of the code in a package could be declarative. (And there could be an option for an ` __init__.jl` file that runs any imperative code that is needed after all of the declarative code has been loaded). This probably wouldn’t improve the discoverability of code dependencies, but then you wouldn’t have to worry about manually ordering `include` statements anymore. Any order of `include` statements would work.
On the other hand, this could make discovering the code dependencies by inspection even harder… 😂
---
### Author:  [@jonathan-laurent](https://discourse.julialang.org/u/jonathan-laurent)
#### Post date: [June 12, 2021, 5:13pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/105 "2021-06-12T17:13:58Z")
> [@CameronBieganek](#):
>
> Just because a particular software engineering practice is “good and established” for some other languages does not mean that it is automatically the best practice for Julia software. Proponents of functional programming like to claim that functional programming obviates the need for all or most of the OOP design patterns advocated by the [Gang of Four](https://en.wikipedia.org/wiki/Design_Patterns). In a similar vein, perhaps multiple dispatch (along with first-class functions and data types that are not bundled with methods) obviates the need for deeply nested modules a la Java and Python?
This is of course a fair point in general. In this argument, I really tried to abstract myself from language-specific debates and what I call “good software engineering practices” could be summarized as _dividing a package into a lot of small, lowly-coupled semantic units with local definitions and a clear dependency graph_. It is my opinion that the current status quo does not make this as easy and natural as it should be.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [June 12, 2021, 5:43pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/106 "2021-06-12T17:43:55Z")
It is a pity the issue [https://github.com/JuliaLang/julia/issues/4600](https://github.com/JuliaLang/julia/issues/4600) features no proper discussion. Any dissenting voice is slapped down (from about a week ago). As @jonathan-laurent pointed out, it would be a good discussion to have. Obviously, with all opinions heard.
Edited: The time allusion was wrong. My sincere apologies to the contributors to this thread.
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [June 12, 2021, 5:46pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/107 "2021-06-12T17:46:41Z")
> [@CameronBieganek](#):
>
> (along with first-class functions and data types that are not bundled with methods) obviates the need for deeply nested modules a la Java and Python?
This seems to be a big component of the trepidation for allowing non-include based (e.g. FromFile style, or whatever syntactic construct shakes out of #4600) code loading. I think it’s good to look at how other kindred languages (i.e. _not_ Java and Python) handle it.
Ruby is perhaps the closest analogue to Julia. `require` is essentially Julia’s `include`, and from what I’ve seen Gems favour pulling require statements closer to the top level where applicable. However, Ruby _also_ has `include`, which functions much like the proposed syntax in #4600. I should note that Ruby allows for re-definition of classes as well.
Has that lead to a phenomenon of catastrophic nesting? Well, if we take a look at a large and mature Ruby codebase (GitLab), almost all files come in at \<= 3 levels of folder nesting and \<= 2 levels of module nesting. I looked at the first 5 large-ish Julia projects that came to mind and found \<= 2 levels of folder nesting with \<= 2 levels of module nesting.
Moving to languages with multiple dispatch, Common Lisp supports code loading via `load` (much like `include`) _and_ `require` (much like `using`/`import`, but will load code if it isn’t already). For comparison, I pulled up the two largest projects I could think of (Clasp and SBCL). Here there was \<= 2 levels of folder nesting and \<= 1 level of module nesting.
Lastly, Clojure is an interesting example because it also has multiple dispatch and enforces a linear include order, but works with the dreaded java namespace system. I looked at 3 projects I knew to be of reasonable complexity (Figwheel, Ring and Neanderthal, representing frontend/tooling, backend and scientific computing respectively). Despite liberal use of `defmethod`, there were \<= 5 levels or folder nesting and \<= 4 levels of namespace nesting.
So in conclusion, allowing for code loading facilities more complex than `include` has not resulted in a modulepocalypse in well-liked, general-purpose languages (i.e. not Matlab) which share some common design philosophy with Julia. Given this, I think it falls on those who do not want any changes to the status quo to provide evidence of (non-contrived) scenarios where allowing something like #4600 would have an outsized negative impact.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [June 12, 2021, 5:52pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/108 "2021-06-12T17:52:40Z")
And conversely, for the proponents to show where the current mechanism fails so badly that something else is needed. And to demonstrate that the benefits outweigh yet another layer of complexity.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [June 12, 2021, 5:56pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/109 "2021-06-12T17:56:20Z")
> [@CameronBieganek](#):
>
> As @jonathan-laurent admitted in his Github post, using a lot of sub-modules adds a lot of boiler plate to a package, since you’re constantly importing and exporting types and functions from various submodules within your package.
Well, imo you should use submodules when the “surface area” of the API the module provides is small. In that case you hide a bunch of implementation from other users of that submodule. If you have to export everything anyway, then yes, using a submodule didn’t really buy you anything. I think Documenter.jl does a good job with submodules.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [June 12, 2021, 6:41pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/110 "2021-06-12T18:41:00Z")
> [@ToucheSir](#):
>
> almost all files come in at \<= 3 levels of folder nesting
Just a point of clarification… I’m not certain which of the following corresponds to 3 levels of folder nesting:
```julia
src/Folder1/Folder2/file.jl
src/Folder1/Folder2/Folder3/file.jl
src/Folder1/Folder2/Folder3/Folder4/file.jl
```
It seems like either one of those could be considered “3 levels of folder nesting”, depending on how we define folder nesting. 🙂
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [June 12, 2021, 7:58pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/111 "2021-06-12T19:58:36Z")
> [@jonathan-laurent](#):
>
> _dividing a package into a lot of small, lowly-coupled semantic units with local definitions and a clear dependency graph_
If we could (edit: we can 🙂) do this I posted in this other thread, at the cost of having a “directory-\>module” correspondence and a Project.toml file for each module, that could be achieved without any new language feature (and from what I see adding the benefit of expanding the possible uses of each module beyond the limited scope of one package:
> [@(not to be taken seriously) ideas about modules and packages](https://discourse.julialang.org/t/not-to-be-taken-seriously-ideas-about-modules-and-packages/62672/6):
>
> I am not sure if almost that is already possible, in fact. If we have a package with a directory structure like: MyPackage --- Project.toml --- src --- MyPackageUtils ------Project.toml ------src ----MyPackageExamples -------Project.toml -------src Can we register the three packages there independently in the general registry? Can two different packages point to different directories of the same GitHub repository? And, of course, have the these packages dependent on each other (I don’t mean ci…
---
### Author:  [@lamorton](https://discourse.julialang.org/u/lamorton)
#### Post date: [June 13, 2021, 2:00am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/112 "2021-06-13T02:00:33Z")
> [@CameronBieganek](#):
>
> > [@jonathan-laurent](#):
> >
> > failing to incentivize good and established software engineering practices for package developers
>
> Just because a particular software engineering practice is “good and established” for some other languages does not mean that it is automatically the best practice for Julia software.
I agree with @CameronBieganek – one shouldn’t cargo-cult best practices, it’s important find out what works best for Julia. “People coming from Python find this unintuitive” is somewhat persuasive based on [Julia user demographics & trends](https://julialang.org/blog/2020/08/2020-julia-user-and-developer-survey/), but a more objective reason would be more compelling.
However, @jonathan-laurent has expressed a pretty convincing downside to the `include` pattern, IMO. The key issue is this awkward ‘M’-shaped access pattern that one has to go through when trying to wrap one’s head around a new codebase. For instance, my first experience with a new code base is when I run a tutorial and I manage to cause an error. When I start debugging, my entry point is generally inside some `included` file, so I have to infer where that got `included`, hop up to that level, then hop down again into some other `included` file, …, to do reaching definition analysis. This seems to be objectively somewhat more difficult than one would like.
I’m trying to understand why `include` seems to break my expectations or intuition for how name resolution works. I think it boils down to the fact that every file in Python also defines a namespace effectively, if I always use `import` and never `exec`. The equivalent in Julia would be putting all the `include`d code within modules inside the files, IIUC.
Here’s a concrete example of what bugs me: suppose I put this in “MyCalc.jl”
```julia
function calculate(x)
x*subcalc(x)
end
```
Then I fire up Julia:
```julia
>subcalc(x) = x^2
>include("MyCalc.jl")
>calculate(2)
8
```
I’m used to the equivalent of this being an error in Python. It would look like:
```julia
def calculate(x):
return x*subcalc(x)
```
and then
```julia
>from calculate import calculate
>def subcalc(2)
> return x**2
>calculate(2)
File "/Users/.../calculate.py", line 10, in calculate
return x*subcalc(x)
NameError: name 'subcalc' is not defined
```
I like the discipline this provides. I have three ways to approach this in Python:
1. Put the two definitions in the same file, when they belong together.
2. Ensure that I import the definition of `subcalc` into `calculate.py` so it gets closed over, when `subcalc` needs to be shared by other modules.
3. Explicit dependency injection: redefine `calculate` to take `subcalc` as a function-type argument, when I want to dynamically change how `calculate` works
What Python doesn’t let me do (w/o `exec`) is what one might call “implicit dependency injection:” where `subcalc` is left undefined inside `calculate.py` and then finds its way in, from the importing context, through the back door. This seems pathological to me, from the perspective about reasoning about the code locally, but maybe it’s b/c I’m used to [thinking in Blub](http://paulgraham.com/avg.html) and haven’t really grokked the implications of multiple dispatch yet.
Three questions I have for more experienced Julia developers:
- Are there _other_ use-case where that “implicit dependency injection” really comes in handy, beyond package structuring?
- For the specific case of package structuring, am I correct in summarizing the arguments against each of the 3 options as:
1. Putting all the code in one file is unergonomic
2. Putting the code into modules inside each file means boilerplate headers and makes refactoring difficult
3. Explicit dependency injection is awkward, because you’d just end up writing a wrapper in `calculate.jl` to close over `subcalc`?
- Are there situations where you would want `calculate` to use use multiple definitions of `subcalc`? (Maybe the answer is ‘always, b/c multiple dispatch’? But I’m not sure if having multiple methods is different from having multiple functions…not a meaningful distinction in Python anyway!)
My gut is telling me that this “implicit dependency injection” capability is a recipe for bugs, because it’s too powerful for the purpose of structuring packages – it seems like you could play some really counterintuitive tricks with it, but it’s intended use case here seems like a small subset of what it could actually be used to do.
[Edit: I can’t count…]
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [June 13, 2021, 3:29am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/113 "2021-06-13T03:29:33Z")
There’s no magical “implicit dependency injection” going on here. All that’s happening is that the code for a single module is split between more than one file. If you were to concatenate the files and include only the concatenated file in the module, the results would be exactly the same.
To use your example, the following two scenarios are exactly equivalent:
### Scenario 1
```julia
module A
function calculate(x)
x*subcalc(x)
end
subcalc(x) = 2x
end
```
### Scenario 2
```julia
# file A.jl
module A
include("calculate.jl")
include("subcalc.jl")
end
```
```julia
# file calculate.jl
function calculate(x)
x*subcalc(x)
end
```
```julia
# file subcalc.jl
subcalc(x) = 2x
```
---
### Author:  [@lamorton](https://discourse.julialang.org/u/lamorton)
#### Post date: [June 13, 2021, 4:40am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/114 "2021-06-13T04:40:48Z")
Does anyone ever write conditional `include` statements? Or nested includes? Or including the same thing in multiple places? If I knew I could assume that a given file was always included exactly once, in the same place, then I would feel like it was equivalent to just copy/pasting the code in. My guess is that people probably don’t do crazy stuff (at least not intentionally!) but my point is that from _within_ the file, there’s no way to tell that it won’t happen. I suppose also that the indirection of the package management system means that people won’t be reaching out to `include` something from inside a package they’re importing, and that the package wraps the `included` code in a module as well, so it’s not just getting dropped “bare” into someone else’s code when they import the package. I think you’re telling me it’s safe to make assumptions I wouldn’t a-priori know that I could make.
I imagine a response to my concern might be: “A bare file like this isn’t meaningful until it is included somewhere - it’s just a half-finished thought. There’s no point trying to reason about it in isolation – stop trying to take that perspective, and just look at the system from the top, where the `include` statement happens. Everything makes sense this way.”
I have two responses to that hypothetical:
- If there is a mechanism in Julia that ensures any given file is included once and only once, then I’m happy, b/c that mechanism can also be used mechanically to locate _where_ this include is happening and take me to it deterministically.
- If included files can’t be understood in isolation, then they shouldn’t be isolated.
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [June 13, 2021, 4:43am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/115 "2021-06-13T04:43:53Z")
Could someone say why it’s useful to have a single module split between multiple files? If a module is a single logical unit, what’s the point of using multiple files? If the other files _aren’t_ part of the same logical unit, I can use multiple submodules.
---
### Author:  [@ChunXi\_Zhang](https://discourse.julialang.org/u/ChunXi_Zhang)
#### Post date: [June 13, 2021, 6:50am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/116 "2021-06-13T06:50:38Z")
I’m a person who used Python heavily and just learn some C++ before using Julia.
To me, Python is a world where you have file/folder as module and package, and import it if you need to use it, no matter it’s your code or others package, you know where the code is coming from within every single file.
People from python is getting used to having A.B in A, so there is something else in A would use A.B.
And C++ people is kind of #include everything, it feels like copy and paste everything into a single file and compile it. There are just files and includes.
Julia’s system is looking similar to C++20’s module system, which is not used by C++ world yet. Plus a package system.
In practice, after reading several matured Julia project, I feel like it’s more of old C++ way for most Julia packages. People just don’t use submodule, their package just has one single module which include all jl files within it. They keep things simple within a package so that any complicated dependencies problem can be handled by `Pkg.jl`.
Even when they do use submodule, like there is CUBLAS within `CUDA.jl`, it’s not like how people from python would think of, where they have something in CUDA.jl which require the submodule CUBLAS, but actually the submodule CUBLAS is quite independent, which using …CUDA, and CUDA would include CUBLAS and export it.
It’s like there is A.B, but it’s just feels like a independent package(actually a module) written in A, but A doesn’t use B at all, it just export it. And everything that is used in A are just files getting included.
This how people’s think A.B is where the source of the confusing for people coming from python.
For example python people would put utils.jl as a module, and it’s feels very bad to have to `include("utils.jl")` and then `using .Utils`, and there maybe several places which use it, so there will be files been included several times.
But every package I saw for Julia, they just have utils.jl as file, not as module, and include(“util.jl”) at the top of the main module.
To me I feel like there is some redundancy to have file/module/package as 3 independent thing, it gives some people freedom but it gives others confusing. And in real world Julia packages, you don’t really see people using module concept efficiently. It’s more of people use files within package, and use package for real dependency problem.
I would suggest to clarify this mindset difference in the [noteworthy-differences](https://docs.julialang.org/en/v1/manual/noteworthy-differences/) part of docs, where there is just one single line mentioned `The logical Julia program structure (Packages and Modules) is independent of the file strucutre (include for additional files), whereas the Python code structure is defined by directories (Packages) and files (Modules).`
People from python don’t understand what it really means for package/file/module as independent things, and how to manage the code in such setting.
---
### Author:  [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)
#### Post date: [June 13, 2021, 10:07am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/117 "2021-06-13T10:07:03Z")
> [@lamorton](#):
>
> Does anyone ever write conditional `include` statements?
Yes, but it’s rather uncommon. I’ve mostly seen it and used it myself in conjunction with the [Requires](https://github.com/JuliaPackaging/Requires.jl) package. Otherwise it should mostly be conditioned on platform or availability of external dependencies.
> Or nested includes?
This is somewhat common for large packages. Especially if submodules are used it’s kinda natural to let those handle the includes of their own files.
> Or including the same thing in multiple places?
This is rare but I’ve seen it in the wild. I’m not convinced it has ever been a good design though.
> I suppose also that the indirection of the package management system means that people won’t be reaching out to include something from inside a package they’re importing
I’ve never seen this and it would be a very questionable thing to do.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [June 13, 2021, 10:09am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/118 "2021-06-13T10:09:42Z")
> [@jzr](#):
>
> why it’s useful to have a single module split between multiple files?
Even if various parts form a coherent whole, it might make sense to split code to multiple files for easier code navigation (ie no distractions from other parts when you are working on something).
This is of course a matter of taste, people should do what they like.
---
### Author:  [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)
#### Post date: [June 13, 2021, 10:41am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/119 "2021-06-13T10:41:51Z")
> [@jzr](#):
>
> Could someone say why it’s useful to have a single module split between multiple files?
For overview and to some extent it can reduce version control conflicts when multiple people are working on the code.
Anecdotally I recently inherited some Python code at work which included a 1600 line class. It didn’t do anything egregious and outsourced a lot of its functionality to other files. It was simply big and had somewhat ambitious docstrings. It was, however, very difficult to get an overview of the code. Eventually I expect I can refactor it to be more manageable but short term I’m rather missing the option to split it into 3 or 4 files (based on sub-functionality) and just include the pieces to stitch them together.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [June 13, 2021, 1:48pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/120 "2021-06-13T13:48:12Z")
> [@lamorton](#):
>
> What Python doesn’t let me do (w/o `exec` ) is what one might call “implicit dependency injection:” where `subcalc` is left undefined inside `calculate.py` and then finds its way in, from the importing context, through the back door. This seems pathological to me, from the perspective about reasoning about the code locally, but maybe it’s b/c I’m used to [thinking in Blub](http://paulgraham.com/avg.html) and haven’t really grokked the implications of multiple dispatch yet.
You’re thinking in Blub 😉
Files have no meaning in the structure of a Julia program, (besides the main PackageName.jl). They are just a way of breaking up code, because 5000 line files are hard to grok, and annoying to use with git.
If your `subcalc` method is defined or imported anywhere in a module then it is available everywhere in the module. There is no “back door” because all the files included in a module are in the same house already. And there is no “implicit dependency injection” happening.
It does work how you want if you use modules, rather than just files. But mostly we don’t do that because 1. we just reuse the same method names everywhere and have few name space collisions. 2. packages are the level of modularity. To illustrate, I have a package dependency depth of five packages that I maintain in some places. But a maximum module depth of 1 inside any of them.
> I suppose also that the indirection of the package management system means that people won’t be reaching out to include something from inside a package they’re importing, and that the package wraps the included code in a module as well, so it’s not just getting dropped “bare” into someone else’s code when they import the package. I think you’re telling me it’s safe to make assumptions I wouldn’t a-priori know that I could make.
Additionally, you _can_ make this assumption a-priori. A file is only dropped bare into your code when you manually call `include`. `import` imports a module, not raw code. Files and modules are not related.
[Previous page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=5)
[Next page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=7)