# \[ANN\] PatModules.jl: a better module system for Julia

**URL:** <https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226>\
**Category:** Package Announcements\
**Tags:** code-organization\
**Created:** [December 22, 2020, 1:09pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226 "2020-12-22T13:09:19Z")\
**Posts on this page:** 1\
**Showing post:** 38

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [December 26, 2020, 2:57am UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/38 "2020-12-26T02:57:14Z")

</div>

Okay, quite a lot to unpack here. Some quotes-with-answers deliberately out of chronological order for better presentation.

> [@kevbonham](#):
>
> I think you mean to say that it’s not organized in a way you find optimal. Surely you understand that saying the code in all the major packages you’ve looked at is low quality is not, in fact, respectful.

> [@johnmyleswhite](#):
>
> I’d suggest all participants take a break on this thread until the New Year.

I’ll start off by apologising if I’ve come across the wrong way. I certainly don’t mean to offend anyone. Clearly I have a controversial opinion – I am trying to express disagreement without derogation.

I’ll restate that for emphasis: I absolutely don’t mean to cause offense.

> [@StefanKarpinski](#):
>
> Welcome, @patrick-kidger! Rather than take the opinion of StackOverflow user395760 as a given, it would be good to understand why the current design is problematic in your view. What concrete issues have you observed it causing?

Thanks for the welcome! Okay, let’s get into the meat of this.

I’m constructing a module/package/some large blob of code.  
I have two files `A.jl` and `B.jl`, which depend upon some common functionality. The typical pattern is to factor this out into some other file, in my case often with an unimaginative name like `utils.jl`.

In order for `A.jl` and `B.jl` to see the definitions of `utils.jl`, they must both `include("utils.jl")`. This poses a problem: they cannot both perform this inclusion. Eventually both `A.jl` and `B.jl` will themselves get included somewhere, and then `utils.jl` has been included twice. The problem with this approach is the _problem of duplication of definitions_.

For example if this occurs within some module hierarchy, then we can end up with two distinct copies of the contents of `utils.jl`, contained within different modules. This isn’t a huge issue if `utils.jl` only defines pure functions, but if `utils.jl` defines some types, with functions dispatching based upon these types, then the copies are mutually unintellegible: you cannot dispatch to functions defined in one copy using the type defined in the other.

The solution is [apparently](https://stackoverflow.com/a/63084743) to include both `A.jl` and `B.jl` in some other file, say `entry_point.jl`, and require that `entry_point.jl` will `include("utils.jl")` on `A.jl` and `B.jl`’s behalf. Indeed this is the standard pattern within several [major projects](https://github.com/SciML/OrdinaryDiffEq.jl/blob/f602235e46fa8c5880f65b677178e901595637e3/src/OrdinaryDiffEq.jl), and I imagine the pattern that most people here are familiar with.

Unfortunately, this has its own problem: `A.jl` and `B.jl` are no longer self-contained. If `A.jl` wishes to use some function `foobar()` defined in `utils.jl`, then it simply uses it without qualification, trusting that it will be made available for it. This is the _problem of not being self contained_, which means that _the dependency structure between files is not made explicit_.  
This implies several problems:

- The code becomes harder to read, and to reason about: each file is implicitly assumed to be executed in some unspecified context.
- It is harder to locate the functionality you are depending upon; as others have noted above this typically requires something like IDE support to track down.
- Additional manual labour is required to ensure that `entry_point.jl` runs its `include`s in the correct order.
- It becomes harder to locate old/dead code that isn’t depended upon by anything.

And moreover these issues are generally exacerbated once multiple developers are involved.

I don’t think these issues are controversial – from earlier in this thread:  
@oxinabox: _“… It’s a fair complaint.”_  
@aplavin: _“one of inconveniences with the current include system is that there is literally no way to tell what are the dependencies of a specific source file”_  
(If either of you feel I’m misrepresenting your point of view here then do please let me know and I’ll take it out.)

So whilst the limitations of this approach are to some degree manageable, they _are_ limitations, and ones with increasing bite as project size grows. It is not overstating my position to say that I think this is the single biggest limitation to work around when using the Julia language; at least that I’m aware of.

As an explicit example, try having a look through the source code for PyTorch. The Python bits (which follow the first pattern) are generally easy to follow. The C++ bits (which follow something akin to the second pattern) are generally difficult to follow.

Do note that ultimately this all an issue about handling files – not modules, nor packages. (Despite the title of this thread – the focus on modules has been because they can be used as a potential solution.)

So what _is_ the solution? (Beyond just putting up with it.) As far as I can tell, until now there hasn’t been one. PatModules.jl is one (deliberately simple) approach, but not one that I’m particularly wedded to. I think if a solution to this problem made its way into the language as a whole I’d probably advocate for a different more sophisticated option. But I shan’t get into that now – let’s focus on establishing whether there is an issue or not first.

Does that all make sense? What are your thoughts?

> [@kevbonham](#):
>
> If I do `using Tables` , and then `using DataFrames` , the later of which also does `using Tables` , there’s no duplication of definitions, is there?

Correct – because both are installed as packages. (In this scenario Julia keeps a global reference of all imported packages and re-uses them if possible.) This discussion / my point is focused solely on the construction of a single package (or more generally some complicated blob of code), and ways to split code across multiple files when doing so.

> [@kevbonham](#):
>
> You might take this as an opportunity to evaluate some of your assumptions. Given your initial statement that you love everything about julia except for this, I take it you recognize the care and thoughtfulness with which the language was designed. It is certainly possible that we all have blinders on and this really is a wart that needs addressing (if so, kudos for trying to address it!). But might it also be possible that there’s something you’re overlooking?

Quite possibly I am wrong. I haven’t been convinced otherwise yet, but I promise you I _am_ reading every reply, and trying not to be a zealot about anything.

> [@kevbonham](#):
>
> I’m struck by the fact that you created your discourse account a week ago and don’t have any other posts asking about how people organize their code, how to avoid duplicate definitions etc.

I spent a fair bit of time searching around looking at existing solutions to this problem, and existing thoughts on how things may be improved:

[Current recommended best practice 1](https://stackoverflow.com/a/63084743)  
[Current recommended best practice 2](https://www.reddit.com/r/Julia/comments/hwxpgm/beginner_question_files_modules_and_include_vs/fz2lzwe/)  
[Current way of performing relative imports](https://discourse.julialang.org/t/julia-relative-imports-issue-4600-exactly-how-not-that-difficult-is-it/34503/4)  
[An example of what is done in existing major packages](https://github.com/SciML/OrdinaryDiffEq.jl/blob/f602235e46fa8c5880f65b677178e901595637e3/src/OrdinaryDiffEq.jl)  
[A comparison to C++ (a language with the same basic issue)](https://github.com/JuliaLang/julia/issues/29966)  
[#4600: a potential change, but not really a fix](https://github.com/JuliaLang/julia/issues/4600#issuecomment-327573232)

With the general overview being that (a) the problem exists, (b) it has already been acknowledged, but (c) there are at present no good solutions.

* * *

Phew, that was a long post. Thank you to those that read it in its entirety.

_PS: And since I didn’t comment on it earlier:_

> [@ToucheSir](#):
>
> PS: Neural CDEs are great 🙂

_Thank you! It’s very flattering to be recognised “in the wild”._

---

_[View the full topic](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226)._