# \[ANN\] PatModules.jl: a better module system for Julia
**URL:** https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226
**Category:** Package Announcements
**Tags:** code-organization
**Created:** [December 22, 2020, 1:09pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226 "2020-12-22T13:09:19Z")
**Posts on this page:** 20
**Page:** 3
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 26, 2020, 9:56am UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/41 "2020-12-26T09:56:20Z")
> [@patrick-kidger](#):
>
> I have two files `A.jl` and `B.jl` , which depend upon some common functionality. The typical pattern is to factor this out into some other file, in my case often with an unimaginative name like `utils.jl` .
>
> In order for `A.jl` and `B.jl` to see the definitions of `utils.jl` , they must both `include("utils.jl")` . This poses a problem: they cannot both perform this inclusion. Eventually both `A.jl` and `B.jl` will themselves get included somewhere, and then `utils.jl` has been included twice. The problem with this approach is the _problem of duplication of definitions_ .
If `A.jl` and `B.jl` are in the _same module_, then the main source file will typically have
```julia
include("utils.jl")
include("A.jl")
include("B.jl")
```
If `A` and `B` are _different modules_, then either the utilities could live in their own module, and both A and B would be `using ThoseUtils`, or alternatively either module could contain those utils the other would be using it.
> [@patrick-kidger](#):
>
> This poses a problem: they cannot both perform this inclusion. Eventually both `A.jl` and `B.jl` will themselves get included somewhere, and then `utils.jl` has been included twice. The problem with this approach is the _problem of duplication of definitions_ .
I still think you are laboring under a misunderstanding: this is not how `include` is used in Julia. You `include` a file within a single module _once_. I would really recommend reading
[https://docs.julialang.org/en/v1.7-dev/manual/modules/](https://docs.julialang.org/en/v1.7-dev/manual/modules/)
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [December 26, 2020, 10:13am UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/42 "2020-12-26T10:13:45Z")
> [@Tero\_Frondelius](#):
>
> > [@aplavin](#):
> >
> > in order to copy a part of code to another project.
>
> This doesn’t sound like the right way of working. You should use `import AwesomeModule: functiontoborrow` syntax instead.
Could you please elaborate a bit how `import` solves the problem?
To give a simple example of what I meant by source files being non-self-contained:
Suppose I’m reading source of `CSV.jl` and come across this line ([CSV.jl/src/detection.jl at 04a2cc7caa7eff226d8dadeae7d63d8d9867a19b · JuliaData/CSV.jl · GitHub](https://github.com/JuliaData/CSV.jl/blob/04a2cc7caa7eff226d8dadeae7d63d8d9867a19b/src/detection.jl#L191)):
`makeunique([normalizenames ? normalizename(x) : Symbol(x) for x in names])`.
Oh, I need something like this in my project, completely independent of CSV! OK, but where do these functions `makeunique` and `normalizename` come from? There are no imports in this file at all - so I don’t even know if these functions are in `CSV.jl` or some other package it depends on. One needs to use github search and hope that it points to the definition of these functions, if they are in `CSV.jl` at all. If they are in another package it’s even more complicated.
Contrast this to each file being a separate “module-like” entity, with imports at the top - something like:
```julia-auto
using Parsers
using PooledArrays
using .utils
# code
```
Easy to guess that those two functions do not live in `Parser` or `PooledArrays`, even if imported names are not listed explicitly. Then no search is required: they are contained in `utils.jl`.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 26, 2020, 10:25am UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/43 "2020-12-26T10:25:57Z")
> [@aplavin](#):
>
> Oh, I need something like this in my project, completely independent of CSV! OK, but where do these functions `makeunique` and `normalizename` come from? There are no imports in this file at all - so I don’t even know if these functions are in `CSV.jl` or some other package it depends on.
Captain Tooling to the rescue!
```julia
julia> using CSV
julia> parentmodule(CSV.makeunique)
CSV
julia> methods(CSV.makeunique)
# 1 method for generic function "makeunique":
[1] makeunique(names) in CSV at /home/tamas/.julia/packages/CSV/la2cd/src/utils.jl:282
julia> parentmodule(CSV.normalizename)
CSV
julia> methods(CSV.normalizename)
# 2 methods for generic function "normalizename":
[1] normalizename(name::Symbol) in CSV at /home/tamas/.julia/packages/CSV/la2cd/src/utils.jl:274
[2] normalizename(name::String) in CSV at /home/tamas/.julia/packages/CSV/la2cd/src/utils.jl:275
```
> [@aplavin](#):
>
> One needs to use github search and hope that it points to the definition of these functions
Or, alternatively, learn about [the relevant tools](https://docs.julialang.org/en/v1/stdlib/InteractiveUtils/).
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [December 26, 2020, 10:57am UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/44 "2020-12-26T10:57:56Z")
Of course, many inconveniences can be reduced with tooling. And github search that I mentioned is a tool as well.
But the process is still more complicated than it could be. That is, one needs to run `julia` + `add CSV#master` + `import CSV` to use julia introspection functions - compared to simply looking up imports at the top of the source file. Moreover, if the package in question has heavy dependencies (e.g. `RCall.jl`), they will be installed as well taking lots of time and bandwidth, even if not really required for the specific function. If the package repo is non-public or requires some kind of proxy to access then even more setup is needed. If package is not compatible with my julia version… Etc.
Note I’m not saying anywhere that it’s _impossible_ to find where a specific function is defined. It’s just significantly less convenient than could be.
---
### Author:  [@cce](https://discourse.julialang.org/u/cce)
#### Post date: [December 26, 2020, 1:09pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/45 "2020-12-26T13:09:40Z")
> [@Tamas\_Papp](#):
>
> Or, alternatively, learn about [the relevant tools](https://docs.julialang.org/en/v1/stdlib/InteractiveUtils/).
I would note that neither `parentmodule` nor `methods` are documented in that link, yet they are the functions you use to make your point. I was unaware of either, thanks. Your amazing efforts on the public list to inform people are really welcome; I wish the documentation included this wisdom. Julia’s documentation could additional attention, based upon common threads in forum discussion. Conversely, the best forum responses are ones that point to clear/concise documentation.
---
### Author:  [@cce](https://discourse.julialang.org/u/cce)
#### Post date: [December 26, 2020, 1:22pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/46 "2020-12-26T13:22:05Z")
> [@patrick-kidger](#):
>
> I’m constructing a module/package/some large blob of code.
> I have two files `A.jl` and `B.jl` , which depend upon some common functionality. The typical pattern is to factor this out into some other file, in my case often with an unimaginative name like `utils.jl` .
```julia
module MyPackage
include("utils.jl")
include("A.jl")
include("B.jl")
end
```
Then in your tests you `import MyPackage` and use the public (exported) or private interfaces as needed. I’ve not had a problem with this common pattern. Sure, it doesn’t encode that A and B are otherwise independent. However, does that really matter from a user’s perspective?
If someone is going to maintain your code, they’ll have to learn the dependency graph anyway; so this particular concern is probably the least of the worries a collaborator may have. For me, the primary challenge I have with maintaining code is the margin. I prefer code to wrap at 76 characters… or less. With 50 year old eyes and having not listened to my mum warning me about staring at the sun when I was 5yo, I need big fonts on a big display… or, better, to send the code to the printer for hard-copy. I notice that those who argue for 132 columns are often quite young with eagle eyes.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 26, 2020, 2:00pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/47 "2020-12-26T14:00:37Z")
> [@cce](#):
>
> neither `parentmodule` nor `methods` are documented in that link, yet they are the functions you use to make your point
I was pointing out additional related tools.
> [@cce](#):
>
> I wish the documentation included this wisdom
This is your lucky day then: `method`s is mentioned in the [Methods chapter](https://docs.julialang.org/en/v1.7-dev/manual/methods/) (among other places), while the [Modules chapter](https://docs.julialang.org/en/v1.7-dev/manual/modules/#Qualified-names) includes an example of `parentmodule`.
> [@cce](#):
>
> Julia’s documentation could additional attention
I certainly agree that the documentation could use additional attention, especially in the sense that more people could just read it.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [December 26, 2020, 2:15pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/48 "2020-12-26T14:15:29Z")
> [@patrick-kidger](#):
>
> Clearly I have a controversial opinion – I am trying to express disagreement without derogation…
>
> I absolutely don’t mean to cause offense.
I really appreciate your engagement in the topic, and I again want to commend your initiative. When things get prickly, it can be easy to disengage.
I don’t have much of technical merit to contribute, I just wanted to highlight that I don’t think your opinion itself is what’s controversial, lots of different coding styles and approaches can coexist happily. That’s one of the great things about this community - very few people are dogmatic about anything!
I think what came across poorly is the implication that we’ve all been doing it wrong/poorly. It’s totally fine to want a different structure - indeed one of the amazing things about the language is that you can make something like `PatModules.jl` to help the language conform to your preferences (and easily share it with others that feel the same way).
---
### Author:  [@paulmelis](https://discourse.julialang.org/u/paulmelis)
#### Post date: [December 26, 2020, 3:54pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/49 "2020-12-26T15:54:51Z")
> [@cce](#):
>
> ```julia
> module MyPackage
> include("utils.jl")
> include("A.jl")
> include("B.jl")
> end
>
> ```
>
> Then in your tests you `import MyPackage` and use the public (exported) or private interfaces as needed. I’ve not had a problem with this common pattern. Sure, it doesn’t encode that A and B are otherwise independent. However, does that really matter from a user’s perspective?
I’m curious about the following if this is the idiomatic case for Julia: how would you set up tests for a subset of what’s in the module, e.g. only for code in A? Importing the full package for tests implies that that module is fully functional (or at least syntactically correct). In the Python equivalent form you can import A as a self-contained module (as it always contains its dependencies) and so can write tests in a more fine-grained manner, independent of the state of the full package.
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [December 26, 2020, 4:30pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/50 "2020-12-26T16:30:08Z")
I do not think it is useful to try to test a part of a package in a way that is independent of the rest of the package working. If some subset of a package is independent of all the rest, then:
1. It could be its own package.
2. It can be an inner module, in a separate file, and you can `include` and `import` it in a test set just for it.
However, even if it is an inner module this does not mean it does not import anything from the parent module or sibling modules (an independent piece of code may be in its own file and module, but being in its own file and module does not mean it is an independent piece of code).
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [December 26, 2020, 4:36pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/51 "2020-12-26T16:36:50Z")
I use this same pattern (i.e., the utility/common/shared code is in files which are included in the “outer module”/“entry file”), but I do some extra encapsulation. Many of the included files have a `module` wrapping all of its code. Consequently, the dependence graph is coded at the top of each file by the import of specific functions and types from sibling modules.
---
### Author:  [@oschulz](https://discourse.julialang.org/u/oschulz)
#### Post date: [December 26, 2020, 4:47pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/52 "2020-12-26T16:47:55Z")
Personally, I found Julia’s reliance on `include` unusual at first for a very modern language - compared to Scala, for example, where the compiler figures dependencies out by itself. But now I’ve come to feel that the “simple” include mechanism actually encourages developers to think more carefully about the structure and inner inter-dependencies of their package(s). I certainly haven’t felt limited by it in any way.
One thing that came to my mind - if Julia should, at some point, become able to pre-compile code in parallel - might we then need a more automatic, graph-based mechanism? I’m not really expert enough regarding the Julia compiler to offer an opinion here, though. However, I would expect that the Julia compiler team has considered this already (and may already have a concept in the drawer for that eventuality?).
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [December 26, 2020, 4:57pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/53 "2020-12-26T16:57:34Z")
Code organization is tough stuff.
---
### Author:  [@doomphoenix-qxz](https://discourse.julialang.org/u/doomphoenix-qxz)
#### Post date: [December 26, 2020, 7:38pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/54 "2020-12-26T19:38:57Z")
> [@cce](#):
>
> > [@patrick-kidger](#):
> >
> > I’m constructing a module/package/some large blob of code.
> > I have two files `A.jl` and `B.jl` , which depend upon some common functionality. The typical pattern is to factor this out into some other file, in my case often with an unimaginative name like `utils.jl` .
>
> ```julia
> module MyPackage
> include("utils.jl")
> include("A.jl")
> include("B.jl")
> end
>
> ```
Contrary to what is suggested here (and not just by cce but by several others in this thread) the OP _is aware_ of this style of code organization, has examined it carefully, and has decided it is insufficient for his purposes. He explained this in his long post above:
> [@patrick-kidger](#):
>
> The solution is apparently to include both `A.jl` and `B.jl` in some other file, say `entry_point.jl` , and require that `entry_point.jl` will `include("utils.jl")` on `A.jl` and `B.jl` 's behalf. Indeed this is the standard pattern within several major projects, and I imagine the pattern that most people here are familiar with.
>
> Unfortunately, this has its own problem: `A.jl` and `B.jl` are no longer self-contained. If `A.jl` wishes to use some function `foobar()` defined in `utils.jl` , then it simply uses it without qualification, trusting that it will be made available for it. This is the _problem of not being self contained_ , which means that _the dependency structure between files is not made explicit_ .
> This implies several problems…
OP has written something that I certainly wished for many times when learning Julia. He has also made a good argument that there’s a real use case for `PatModules.jl`: When you are developing a large package and you write functionality (like `utils.jl`) used in several places in your package, but for whatever reason, don’t want to split `utils.jl` into its own package, and yet you want to be clear _where the functions in utils.jl come from so they don’t appear as random names all over your package_.
Even if this is not a reasonable thing to want, the OP wanted it _and_ went to the trouble to implement it himself before telling us about it, and is not forcing anyone to use it. Perhaps his initial tone was not optimal, but he has tried to clarify that he’s not trying to insult anyone. The great thing about open source software is that it allows people to write whatever they want, and the great thing about Julia is that it makes this sort of experimentation easier (can you imagine rewriting Python `import` logic in a package?).
@patrick-kidger I say well done, and time will tell if this package provides a sufficiently elegant solution to a sufficiently common problem that it gains more traction. I wouldn’t give too much weight to the initial reactions here; just keep making `PatModules.jl` awesome. If it’s awesome enough, people will use it.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [December 26, 2020, 7:56pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/55 "2020-12-26T19:56:22Z")
I think not distinguishing between organization of code into **files** and organization of code into **modules** is obscuring the real issues in this discussion. My code is usually divided into modules, with one file per module. I haven’t found any use for multiple files per module or vice versa.
I still don’t see what having the common functionality in the file utils.jl may mean that would necessitate additional structure in addition to what Julia already provides.
---
### Author:  [@Skoffer](https://discourse.julialang.org/u/Skoffer)
#### Post date: [December 26, 2020, 10:06pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/56 "2020-12-26T22:06:43Z")
I think your example shows what is the real problem here. I think, there are two sides to this problem: those who write long and complicated packages and those who want to somehow extend these packages.First group is happy with the structure “one file that includes everything else and multiple other files which presume that context is already defined” and other group vote that such structure leads to worse discoverability, cause you literally need to refer constantly to this “table of contents” and apply some guess work where what is defined.
Now, regarding your question, developers can already do “discoverable” file structure, for example by writing at the beginning of each file, something like
```julia
# used utils: makeunique, normalizename
```
Note comment line here. This approach solve the issue of discoverability (where functions are coming from) without breaking any code at all, but at the price of extra work of adding these verbose explanations. Of course usual developer wouldn’t go that far and make such a comment.
From this point of view, I think it’s easier to understand why proposed file organization doesn’t look appealing. Instead of adding commented lines of function origin, language or package forces them on developer with only benefit that they may omit big “table of contents” file. It doesn’t look like a fair deal: more work with no real benefits except of better discoverability for possible outside developers.
As an extreme example, imagine that you want to go matlab way and put each function in a separate file. Imagine amount of work to make each and every file self-contained with complete context. And imagine amount of work that you’ll have to do if for some reason you change the name of one of the files…
With that said, problem of discoverability do exists. Maybe it is possible to extend Documenter.jl or some other similar package to build automatically something analogous to sitemap only for package, so you can generate html/md/toml file where each and every function can be traced to the file where it is defined.
---
### Author:  [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)
#### Post date: [December 26, 2020, 10:09pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/57 "2020-12-26T22:09:56Z")
I can’t be 100% sure because I am not looking at it from the same vantage point as you @patrick-kidger, but your thought pattern & solutions remind me alot of my own confusion when I started making modules/packages/namespaces in Julia
> [@patrick-kidger](#):
>
> Fundamentally, you have to deal with issues of files being included multiple times. Solutions to this problem (only including files once; using header guards; only including at the top-level) makes tracking dependencies harder to reason about (only including once), entail extra maintenance work (header guards), or pollute namespaces (including at top-level).
### Header guards, as an example
“using header guards” is one of those things that got to me too - coming from a C/C++ background.
### C/C++ vs Julia
But there are huge differences with `include()` in Julia: You don’t include header files in Julia! In C/C++, header files are used to tell the compiler how much space to allocate to objects, where data offsets are, and where to push data on the call stack before executing a function call.
That’s because C/C++ compiles machine code only, and doesn’t retain a table of that information anywhere (At least I don’t think it does when you turn off the debugger options).
Julia, on the other hand keeps this information around when it “compiles” its code (so I guess it does more than just compile). And this is a _ **big** _ difference. It’s also why Julia can do introspection so well (you can even access the original function code if you want).
### When to use Julia `include()`s
First of all, you don’t `include()` external packages. You should _ **only** _ `include()` files that are in your current project or package (baring exceptional, possibly questionable circumstances). When you want to use external packages, you need to use `import` or `using` (but I won’t talk about those right now).
So right there, this is different from C/C++. Whereas in C/C++ you include header files that are built independently of your current project, in Julia, you _ **only** _ should be including code/files that you _ **know** _ are _ **directly** _ part of your project/package.
That means you can _ **guarantee** _ there is no double inclusion because you, as the developer, have control over this entire project/package codebase. Moreover, your code files need _ **only** _ be included once (it is not the same thing as C/C++ headers). That’s why you’ve noticed people often `include()` a bunch of subfiles from the same master file. Once `include()`-ed, that code is available to _ **any other module** _ loaded by Julia (There is no information hiding like there is in C/C++).
### So how do Julia `include()`-s map to C/C++?
In reality, a Julia `include()` statement probably relates more closely to using the C/C++ linker:
```julia-auto
g++ -o myexecutable first.o second.o third.o
```
In Julia, this would look like:
```julia-auto
#myexecutable.jl
include("first.jl")
include("second.jl")
include("third.jl")
...
```
That’s because the Julia interpreter/compiler actually loads that code into memory and is ready to use it as soon as it processes the `include()` statement. It is _ **not** _ a preprocessor directive like it is in C/C++.
### So how do you call code from another module, then?
Well, unlike C/C++, you don’t need to read in a header file to execute code, or build new `structs` in Julia. Once the code is loaded through an `include()` statement somewhere, your code can just execute it directly as long as it knows what the `module` path is:
```julia-auto
#MyProject/src/subfile1.jl
#Why not define a submodule here? It doesn't have to the same name as the file.
#"module"s are really just namespaces, and are not tied to files in any way.
module SubA
module B #Again, why not another namespace here?
struct MyStruct
x::Int; y::Int
end
end #module B
function dosomething(obj::B.MyStruct)
#Do something
end
end #module SubA
#This function will be in the same namespace ("module") as whatever code
#called include("subfile1.jl"):
function dosomethingelse(obj::SubA.B.MyStruct)
#Do something
end
```
* * *
```julia-auto
#MyProject/src/MyProject.jl
#Julia projects & packages need to declare a module (namespace)
#with the same name as the project to function correctly. Julia also
#expects you to create a file under src/ that has the same name
#as the project (thus /src/MyProject.jl).
module MyProject
include("subfile1.jl")
#Create a global object to store state:
glb_obj = SubA.B.MyStruct(3,5)
#Call functions that might have been written/loaded in other files:
SubA.dosomething(glb_obj)
dosomethingelse(glb_obj)
#...
end #module MyProject
```
### Is that the same way we load external packges?
No, not exactly, code from external packages (even if not registered in Julia’s “General” registry) should be _ **indirectly** _ included with either the `using` or `import` statements:
```julia-auto
#MyProject/src/MyProject.jl
#Julia projects & packages need to declare a module (namespace)
#with the same name as the project to function correctly. Julia also
#expects you to create a file under src/ that has the same name
#as the project (thus /src/MyProject.jl).
module MyProject
using CSV
CSV.dosomething() #FYI: Doesn't actually exist.
#...
end #module MyProject
```
Note that Julia ensures that only _ **a single instance** _ of `CSV` is loaded - no matter how many modules (namespaces) call “`using CSV`”. It also ensures that ALL modules calling `using CSV` get a local pointer to the same `CSV` `module` (namespace), which gets loaded _ **exactly once** _ into memory (unless it needs to be re-evaluated for some reason - yeah. there are a bunch of exception cases, sorry.).
---
### Author:  [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)
#### Post date: [December 26, 2020, 10:26pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/58 "2020-12-26T22:26:58Z")
> [@patrick-kidger](#):
>
> I spent a fair bit of time searching around looking at existing solutions to this problem, and existing thoughts on how things may be improved:
>
> [Current recommended best practice 1](https://stackoverflow.com/a/63084743)
> [Current recommended best practice 2](https://www.reddit.com/r/Julia/comments/hwxpgm/beginner_question_files_modules_and_include_vs/fz2lzwe/)
> [Current way of performing relative imports](https://discourse.julialang.org/t/julia-relative-imports-issue-4600-exactly-how-not-that-difficult-is-it/34503/4)
> [An example of what is done in existing major packages](https://github.com/SciML/OrdinaryDiffEq.jl/blob/f602235e46fa8c5880f65b677178e901595637e3/src/OrdinaryDiffEq.jl)
> [A comparison to C++ (a language with the same basic issue)](https://github.com/JuliaLang/julia/issues/29966)
> [#4600: a potential change, but not really a fix](https://github.com/JuliaLang/julia/issues/4600#issuecomment-327573232)
>
> With the general overview being that (a) the problem exists, (b) it has already been acknowledged, but © there are at present no good solutions.
I suggest looking at the following as well:
→ [Dependencies of src files inside a package](https://discourse.julialang.org/t/dependencies-of-src-files-inside-a-package/43144)
I also creating a doc [PR](https://github.com/JuliaLang/julia/pull/37167) to address this issue (I don’t suggest reading the PR thread because it’s a bit hard to read).
Instead, I’ll insert the PR text directly here:
# Inline text from [PR](https://github.com/JuliaLang/julia/pull/37167)
### Julia ⇔ C/C++: Namespaces
- C/C++ `namespace`s correspond roughly to Julia `module`s.
- There are no private functions/variables/modules/… in Julia. Everthing is accessible
through fully qualified paths (or relative paths, if desired).
- `using MyNamespace::myfun` (C++) corresponds roughly to `import MyModule: myfun` (Julia).
- `using namespace MyNamespace` (C++) corresponds roughly to `using MyModule` (Julia)
- In Julia, only `export`ed symbols are made available to the calling module.
- In C++, only elements found in the included (public) header files are made available.
- Caveat: `import`/`using` keywords (Julia) also _load_ modules (see below).
- Caveat: `import`/`using` (Julia) works only at the global scope level (`module`s)
- In C++, `using namespace X` works within arbitrary scopes (ex: function scope).
### Julia ⇔ C/C++: Module loading
- When you think of a C/C++ “ **library** ”, you are likely looking for a Julia “ **package** ”.
- Caveat: C/C++ libraries often house multiple “software modules” whereas Julia
“packages” typically house one.
- Reminder: Julia `module`s are global scopes (not necessarily “software modules”).
- **Instead of build/`make` scripts** , Julia uses “Project Environments” (sometimes called
either “Project” or “Environment”).
- Build scripts are only needed for more complex applications
(like those needing to compile, or download C/C++ executables 🙂 ).
- C/C++ code typically target more conventional applications, whereas Julia
“Project Environments” provide a set of packages to experiment with particular problem
spaces. Julia users typically use problem-specific “scripts” for this type of experimentation.
- To develop a “conventional” application/project in Julia, you can initialize its root directory
as a “Project Environment”, and house application-specific code/packages there.
This provides good control over project dependencies, and future reproducibility.
- Available packages are added to a “Project Environment” with the `pkg> add` tool
(This does not **load** said package, however).
- The list of available packages (direct dependencies) for a “Project Environment” are
saved in its `Project.toml` file.
- The _full_ dependency information for a “Project Environment” is auto-generated & saved
in its `Manifest.toml` file.
- Packages (“software modules”) available to the “Project Environment” are loaded with
`import` or `using`.
- In C/C++, you `#include ` to get object/function delarations, and link in
libraries when you build the executable.
- In Julia, whatever is loaded is available to _all other_ loaded modules through its
fully qualified path (no header file required).
- Use `import SomePkg: SubModule.SubSubmodule` (Julia) to access package submodules.
- **Directory-based package repositories** (Julia) can be made available by adding repository
paths to the `Base.LOAD_PATH` array.
- Packages from directory-based repositories do not require the `pkg> add` tool prior to
being loaded with `import` or `using`. They are simply available to the project.
- Directory-based package repositories are the **quickest solution** to developping local
libraries of “software modules”.
### Julia ⇔ C/C++: Assembling modules
- In C/C++, `.c`/`.cpp` files are compiled & added to a library with build/`make` scripts.
- In Julia, `import [PkgName]`/`using [PkgName]` statements load `[PkgName].jl` located
in a package’s `[PkgName]/src/` subdirectory.
- In turn, `[PkgName].jl` typically loads associated source files with calls to
`include "[someotherfile].jl"`.
- `include "./path/to/somefile.jl"` (Julia) is very similar to
`#include "./path/to/somefile.jl"` (C/C++).
- However `include "..."` (Julia) is not used to include header files (not required).
- **Do not use** `include "..."` (Julia) to load code from other “software modules”
(use `import`/`using` instead).
- `include "path/to/some/module.jl"` (Julia) would instantiate multiple versions of the
same code in different modules (creating _distinct_ types (etc.) with the _same_ names).
- `include "somefile.jl"` is typically used to assemble multiple files _within the same
Julia package_ (“software module”). It is therefore relatively straightforward to ensure
file are `include`d only once (No `#ifdef` confusion).
### Julia ⇔ C/C++: Module interface
- C++ exposes interfaces using “public” `.h`/`.hpp` files whereas Julia `module`s `export`
symbols that are intended for their users.
- Often, Julia `module`s simply add functionality by generating new “methods” to existing
functions (ex: `Base.push!`).
- Developers of Julia packages therefore cannot rely on header files for interface
documentation.
- Interfaces for Julia packages are typically described using docstrings, README.md,
static web pages, …
- Some developers choose not to `export` all symbols required to use their package/module.
- Users might be expected to access these components by qualifying functions/structs/…
with the package/module name (ex: `MyModule.run_this_task(...)`).
### Julia ⇔ C/C++: Quick reference
| Software Concept | Julia | C/C++ |
| --- | --- | --- |
| unnamed scope | `begin` … `end` | `{` … `}` |
| function scope | `function x()` … `end` | `int x() {` … `}` |
| global scope | `module MyMod` … `end` | `namespace MyNS {` … `}` |
| software module | A Julia “package” | `.h`/`.hpp` files
+compiled `somelib.a` |
| assembling
software modules | `SomePkg.jl`: …
`import subfile1.jl`
`import subfile2.jl`
… | `$(AR) *.o` ⇒ `somelib.a` |
| import
software module | `import SomePkg` | `#include `
+link in `somelib.a` |
| module library | `LOAD_PATH[]`, \*Git repository,
\*\*custom package registry | more `.h`/`.hpp` files
+bigger compiled `somebiglib.a` |
\* The Julia package manager supports registering multiple packages from a single Git repository.
\* This allows users to house a library of related packages in a single repository.
\*\* Julia registries are primarily designed to provide versionning & distribution of packages.
\*\* Custom package registries can be used to create a type of module library.
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [December 27, 2020, 12:46pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/59 "2020-12-27T12:46:44Z")
Thanks for sharing your view on this.
> [@Skoffer](#):
>
> Now, regarding your question, developers can already do “discoverable” file structure, for example by writing at the beginning of each file, something like
>
> ```julia
> # used utils: makeunique, normalizename
>
> ```
Comments can and will easily become obsolete, as they are not checked by the compiler at all. Thus one cannot rely on them.
And I didn’t propose to list all names in every import, just the opposite! Even without such lists it’s pretty obvious that `makeunique` and `normalizename` likely lie in `.utils`, and not in `Parsers`.
> [@Skoffer](#):
>
> As an extreme example, imagine that you want to go matlab way and put each function in a separate file. Imagine amount of work to make each and every file self-contained with complete context.
But why would you want to go “matlab way” in julia? Typically there are many short functions, and it’s just crazy to put each into separate file.
Also, my previous message here presented a look from one PoV: as a user who reads package code and wants to understand something there. Below is a view of me a “package developer”.
Take for example a package of mine: [Alexander Plavin / SquashFS.jl · GitLab](https://gitlab.com/aplavin/SquashFS.jl). The source file `api.jl` contains main functions that are supposed to be called by users - the package public API. E.g. `SquashFS.open`, `SquashFS.readdir`, etc. Other files, such as `utils.jl` and `sqfs_structs.jl`, contain lower-level functions that are supposed to be internal.
Initially (and currently) I follow the typical/suggested julian approach: don’t make any separate modules within the package. But this means all the internal functions are available as `SquashFS.xxxxx` and show in autocomplete - confusing users. If every source file was implicitly “module-like” I would just put something like `using .api` into main `SquashFS.jl` file. Internal functions would then automatically be accessible as `SquashFS.utils.xxxxx`.
As this is not the case, I’m thinking of creating a submodule like `Internal` manually, but it’s not really clear for me where to define it. And as I understand from this thread, submodules within a single package are actually discouraged…
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 27, 2020, 1:39pm UTC](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/60 "2020-12-27T13:39:40Z")
> [@aplavin](#):
>
> And as I understand from this thread, submodules within a single package are actually discouraged…
They are perfectly fine. Segregating code into _lots_ of submodules _may_ be a bad idea, and submodules that are usable elsewhere might be better as packages, but a single `Internal` submodule for namespace purposes is not unreasonable.
> [@aplavin](#):
>
> As this is not the case, I’m thinking of creating a submodule like `Internal` manually, but it’s not really clear for me where to define it.
Just create a file `Internal.jl` with `module Internal ... end` that `include`s your other files. then in your main `SquashFS.jl` file do `include("Internal.jl")` and access its names as `Internal.foo` (or do `using .Internal` to get any exports).
[Previous page](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226.md?page=2)
[Next page](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226.md?page=4)