# Single module vs. submodules in a project

**URL:** <https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361>\
**Category:** New to Julia\
**Tags:** performance, modules, code-organization\
**Created:** [February 15, 2021, 11:23pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361 "2021-02-15T23:23:33Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![erans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erans/32/22032_2.png) [@erans](https://discourse.julialang.org/u/erans)\
**Post date:** [February 15, 2021, 11:23pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/1 "2021-02-15T23:23:33Z")

</div>

I have been reading a fair amount (a few references at end of post) about how to “correctly”, or better yet idiomatically, structure code in Julia. In general, there seems to be a strong sentiment that many submodules probably means the code could be better split into several packages.

**My question is** : wouldn’t breaking code into submodules help in terms of precompilation? (assuming that most of the submodules are not constantly updated).

I should note that I don’t fully understand all the intricacies of JIT compilation. It’s therefore quite possible that my assumptions below are incorrect and in that case cool! no submodules for me.  
My sense is that the reason to use Julia is to leverage these sort of subtleties, which is where the performance gains are achieved, so I want to make sure I’m doing it right 🙂

### Example

Say I have a package `MyPkg`:

```julia-auto
Module MyPkg
include("A.jl")
include("B.jl")
end

#contents of A.jl
foo_a(x) = x + 2
[...]
#contents of B.jl
foo_b(x) = 2*fooA(x)
[...]

```

Now let’t assume that:

1. `fooA` (and the contents of `A.jl` is unlikely to change very frequently, while
2. I’m still tweaking `fooB` and might even want to add a few more things.
3. Finally, the contents of `A.jl` have no real meaning on their own (Silly example above aside, maybe they set up some specialized Types for the calculations coming in `B.jl`, are utility functions that are tailored to the functions in `B.jl`, etc.)

_My current understanding_ is that in the construction above any change in `B.jl` would require precompilation of `MyPkg` (upon restarting the REPL), which would also necessarily include the contents of `A.jl`.

I _could_ restructured the code as follows:

```julia-auto
Module MyPkg
include("A.jl")
include("B.jl")
end

#contents of A.jl
Module A
export fooA
fooA(x) = x + 2
[...]
end
#contents of B.jl
Module B
using .A
fooB(x) = 2*fooA(x)
[...]
end

```

If I now make a change to `B.jl` wouldn’t only `MyPkg` and `B` need to be recompiled, while the already precompiled `A` can be used as is?

### Some references

Beyond the Julia documentation (specifically [Modules](https://docs.julialang.org/en/v1/manual/modules/) and [Code Loading](https://docs.julialang.org/en/v1/manual/code-loading/)) and the Pkg documentation, there are also several interesting discussions on this forum about the question (many quite recent):

> [@Proper way of organizing code into subpackages](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/11):
>
> Can you give us a hint of the domain of your application? With more context, it’s possible that we might be able to recommend an alternative way of organizing your code.

> [@Best way to structure Julia code](https://discourse.julialang.org/t/best-way-to-structure-julia-code/26591):
>
> Dear all, I am a bit sad that it was told here on the forum that adding a custom directory to the package search path would be depreciated. If I write a Julia program I would like to structure it by using a module for each separate piece of code. But I want to have all my code in one git repository, and not one repo for each module. Should every module be a package that lives in a separate repo? I think that would be stupid, because it causes lots of unnecessary overhead. What do you think? …

> [@\[ANN\] PatModules.jl: a better module system for Julia](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226):
>
> Hey folks. I’m new to Julia, and loving every aspect of it… except for ways of splitting code over multiple files and folders. There, I swiftly discovered that existing ways just don’t really seem to cut it. Julia primarily uses include(...) to get at code in other files, which [everyone with an opinion on language design seems to agree that you shouldn’t do](https://stackoverflow.com/questions/13570947/what-is-the-difference-between-import-and-include-choices-in-language-design). Fundamentally, you have to deal with issues of files being included multiple times. Solutions to this problem (only including files once;…

> [@How to use local module within a Project directory?](https://discourse.julialang.org/t/how-to-use-local-module-within-a-project-directory/26358):
>
> When I’m not in a Project directory, I can make a file like so: # M1.jl module M1 export M1Type struct M1Type data::Real end end and then from that directory: push!(LOAD\_PATH, pwd() and then using M1 and that all works just fine. But if I’m in a Project directory (I have a Project.toml file), I can’t figure out how to use M1 in a similar way. julia\> using M1 ERROR: ArgumentError: Package M1 not found in current path: - Run `import Pkg; Pkg.add("M1")` to install the M1 …

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 16, 2021, 3:05pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/2 "2021-02-16T15:05:13Z")

</div>

> [@erans](#):
>
> how to “correctly”, or better yet idiomatically, structure code in Julia

You may not need to worry about this unless your codebase is relatively large. Up to 5–10 kLOC (and that’s a _lot_ of code in Julia), a single module/package often works fine.

> [@erans](#):
>
> wouldn’t breaking code into submodules help in terms of precompilation?

I am not sure if there is anything that should have this effect. If compilation time is a concern, just investigate and fix that directly. These excellent blog posts are a good starting point:

> **[Analyzing sources of compiler latency in Julia: method invalidations](https://julialang.org/blog/2020/08/invalidations/)**
>
> Julia runs fast, but suffers from latency due to compilation. This post analyzes one source of excess compilation, tools for detecting and eliminating its causes, and the impact this effort has had on latency.

> **[Tutorial on precompilation](https://julialang.org/blog/2021/01/precompile_tutorial/)**
>
> Tutorial on precompilation

> **[Profiling type-inference](https://julialang.org/blog/2021/01/snoopi_deep/)**
>
> Profiling type-inference

---

<div class="post-metadata">

**Author:** ![jebej](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jebej/32/1784_2.png) [@jebej](https://discourse.julialang.org/u/jebej)\
**Post date:** [February 16, 2021, 4:06pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/3 "2021-02-16T16:06:08Z")

</div>

> [@erans](#):
>
> If I now make a change to `B.jl` wouldn’t only `MyPkg` and `B` need to be recompiled, while the already precompiled `A` can be used as is?

I’m not 100% sure, but I don’t think so. As far as I understand, the precompilation unit is the “package”, so if anything changes within the package, even if in a submodule, the whole package is recompiled.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [February 16, 2021, 5:23pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/4 "2021-02-16T17:23:50Z")

</div>

### Submodules → Several Packages

> [@erans](#):
>
> In general, there seems to be a strong sentiment that many submodules probably means the code could be better split into several packages.

I think you are correct in that the message being sent out in the community does sound the way you summarized it. However, I believe it stems from our inability to concisely express ourselves.

- Software designers use modules as a means to encapsulate a software component, and hide implementation details. Julia doesn’t really do information hiding, so you need more explicit means of communicating your public interface to the user, as well as how to use your software component effectively.
- With most programming languages, module encapsulation is typically done (in part) by wrapping code from each “module” in its own namespace. This ensures functions/variables/constants… will not collide with those of the outside world.
- In Julia, you instead leverage multiple dispatch, by judiciously writing function signatures. Thus, with multiple dispatch, you worry more about _ **method** _ collisions instead of _ **function name** _ collisions.
  - Before namespaces and multi-dispatch, you would have to write something like `stackpush_i64(list,val)`, and `stackpush_f64(list,val)`, etc. to avoid name collisions.
  - After namespaces, you could write `datastructures::i64stack::push(list, val)` to access the push function of `Int64` stack object found in the `datastructures` library. This way, all stacks could share the same name for their interface without worrying about name collisions.
  - Similarly, in object-oriented languages, you can create a stack “object” with `mystack=datastructures::i64stack::stack()`, then add to it using `mystack.push(val)`.
  - In Julia, generic (parameterized) stack objects basically come for free, and the programmer has very intuitive facilities to communicate type information to the compiler (don’t need the intricate tree of namespaces found in other paradigms).

- As you might have noticed, the advantage of having a flatter namespace hierarchy is that your users of your package don’t have to fully qualify `the.path.to.your.sub.module` (which can often make things more readable).
- In Julia, you still have to worry about name collisions for variables and constants, though:
  - But since developers have full control over the entire package they are building/modifying, they can typically manage name collisions effectively.
  - At times, it might still be very practical to create “sub-`module`s” (different namespaces) to deal with these collisions and/or collect software components under a meaningful sub-`module` name. So I think this is where the community’s message about sub-`module`s can get misinterpreted.

I’m still working out how to effectively communicate more optimal ways to leverage Julia’s facilities (Not to mention that I’m still figuring it out). That being said, here are a few tips on how I structure my Julia code:

- I no longer break out different software components to separate “`module`s” (because Julia `modules` are really just namespaces).
- I still try to write out different software components to separate files/folders so I/other developers can easily locate them.
- I only create Julia packages when I want to “bundle” up a collection of software components that are generic enough to be reusable by _ **more than one project/application** _.
- Again: If a software component is not to generic enough to be used by other projects/applications, then it simply exists as separate files/folders, which are directly `include()`-ed in the project/application requiring it. The idea is to help other developers/myself understand the logical structure of the software, and find the relevant code.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [February 16, 2021, 5:49pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/5 "2021-02-16T17:49:55Z")

</div>

> [@erans](#):
>
> wouldn’t breaking code into submodules help in terms of precompilation? (assuming that most of the submodules are not constantly updated).

As @jebej mentionned, I’m pretty certain the precompilation unit is the “package”, not the “module”.

That being said, from a practical standpoint, I do like your idea of splitting out code that seldomly changes, but takes alot of time to compile (assuming that this really is the case).

- The only adjustment I would make is to write it to a separate package (not a submodule).

Note that you might eventually want to merge the two packages back together to avoid registering two modules (and avoid dealing with two separate .git repositories to house your code). I say this because Julia packages are registered a .git repository level (sort of; there are workarounds here, but I think they need a bit of streamlining).

### Pitfalls to precompilation

To take _ **full advantage** _ of precompilation though, you might need to do a bit of research:

- I think some portions of precompilation only gets triggered when a user of your package tries to call a specific `method` in your package (ie calls a function, but for a specific set of types).
- The good news is that I’m pretty certain there are ways to trigger the compilation of these specific methods using [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl) so that it gets included in the precompilation-image of your base package.

### Visibility issue with this thread

You might also want to:

- Check out discourse archives for more info on precompilation.
- Possibly change the title of this thread.

Because it sounds to me you are really looking for help with precompilation, but it’s not obvious from the title.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [February 16, 2021, 5:58pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/6 "2021-02-16T17:58:57Z")

</div>

Oh, and I realize that my suggestion of splitting out code that _ **won’t be used by any other project/package/application** _ goes directly against the post I wrote immediately above it.

In this case however, you have a _ **practical** _ reason to do this (reduce development time), despite it being unnecessary from a _ **structural** _ standpoint for code organization.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [February 16, 2021, 6:26pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/7 "2021-02-16T18:26:41Z")

</div>

> [@MA\_Laforge](#):
>
> As @jebej mentionned, I’m pretty certain the precompilation unit is the “package”, not the “module”.

~~Looking at the contents of `~/.julia/compiled/v1.5` the results are apparently stored per module. And a package as far as I can see is nothing more than a module structured in a specific way. I would even venture (but don’t know) the compiler doesn’t distinguish between the two and only Pkg.jl is aware of packages.~~

 

~~Anyway, did some testing:~~

Edit: redid the checks for caching of the compiled files  
Edit 2: per comment [Single module vs. submodules in a project - #8 by jebej](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/8) below none of the above makes any sense. A real submodule uses a relative form of `using` and then will not show up as a separate cached file.  
Edit 3: deleted the example code which was showing the wrong thing, as Discourse doesn’t support strikethrough of code it seems (even though the edit preview did show it)

---

<div class="post-metadata">

**Author:** ![jebej](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jebej/32/1784_2.png) [@jebej](https://discourse.julialang.org/u/jebej)\
**Post date:** [February 16, 2021, 7:29pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/8 "2021-02-16T19:29:26Z")

</div>

You are not using submodules here, you are just creating two individual “packages”, though because you are pushing to `LOAD_PATH` these things are less clear.

A submodule must be defined within the scope of a parent module, like in the example [here](https://MySubModule) copied below, or by `includ`ing the file containing the submodule within the parent module. The module blocks must be nested.

```julia
module Parent

module Utils
...
end

using .Utils

...
end

```

If you want to do that precompilation test properly with submodules, you would need the following:

```julia
module MyModule # MyModule.jl 

include("MySubModule.jl")
using .MySubModule # note the dot

export f

f(a) = 2*g(a)

end

```

As a side note, pushing to the `LOAD_PATH` is not the best way of organizing code, since you won’t be making use of dependency resolution and reproducible environments you can do with a `Project.toml` file and using `Pkg`.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [February 16, 2021, 8:11pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/9 "2021-02-16T20:11:20Z")

</div>

> [@jebej](#):
>
> You are not using submodules here, you are just creating two individual “packages”, though because you are pushing to `LOAD_PATH` these things are less clear.

Doh! And it even works the same in Python, so silly of me not to realize that. I’ll ammend the comment above.

---

<div class="post-metadata">

**Author:** ![erans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erans/32/22032_2.png) [@erans](https://discourse.julialang.org/u/erans)\
**Post date:** [February 16, 2021, 8:15pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/10 "2021-02-16T20:15:40Z")

</div>

Thanks everyone for the input. This has cleared several things up for me.

From looking at the results in the `~/.julia/compiled/v1.5/` folder I can see that indeed submodules are _not_ precompiled separately. @paulmelis thank you very much for taking the time and effort to run that test. I believe @jebej is correct though, that the issue is that you are using `LOAD_PATH` (btw, avoiding load path is the primary reason why I’ve fallen into this rabbit hole)

> [@Tamas\_Papp](#):
>
> These excellent blog posts are a good starting point:

These are great! and gives me a lot to think about w.r.t precompilation

> [@MA\_Laforge](#):
>
> Because it sounds to me you are really looking for help with precompilation, but it’s not obvious from the title.

Thanks for all the tips, regarding this specific point though, my main interest _currently_ is the best way to structure code. My thought was, that the “best” way would likely best leverage Julia’s capabilities. As far as I can tell, the two main Julia highlights are JIT and multiple dispatch. With that in mind, I was/am trying to figure out how my code structure can best play nice with these concepts.

Thanks again to everyone for the good feedback!

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [February 16, 2021, 8:16pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/11 "2021-02-16T20:16:23Z")

</div>

> [@erans](#):
>
> I believe @jebej is correct though, that the issue is that you are using `LOAD_PATH` (btw, avoiding load path is the primary reason why I’ve fallen into this rabbit hole)

Yep, see my updated comment above

---

<div class="post-metadata">

**Author:** ![erans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erans/32/22032_2.png) [@erans](https://discourse.julialang.org/u/erans)\
**Post date:** [February 16, 2021, 8:18pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/12 "2021-02-16T20:18:13Z")

</div>

woops, I must have been a minute too quick with my post.

---

<div class="post-metadata">

**Author:** ![jebej](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jebej/32/1784_2.png) [@jebej](https://discourse.julialang.org/u/jebej)\
**Post date:** [February 16, 2021, 8:26pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/13 "2021-02-16T20:26:07Z")

</div>

> [@erans](#):
>
> Thanks for all the tips, regarding this specific point though, my main interest _currently_ is the best way to structure code. My thought was, that the “best” way would likely best leverage Julia’s capabilities. As far as I can tell, the two main Julia highlights are JIT and multiple dispatch. With that in mind, I was/am trying to figure out how my code structure can best play nice with these concepts.

As evidenced by the multiple posts on this issue, code organization is quite a tricky thing. I think many people were using the pre 1.0 way to simply have code on the `LOAD_PATH`, and never transitioned to the new method (I know I didn’t get it for a while!).

It is a little more work and complexity, since now individual packages must have a `Project.toml`, and must be `Pkg.dev`ed to the global environment if you want to simply type `use MyPackage` after starting julia.

---

<div class="post-metadata">

**Author:** ![erans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erans/32/22032_2.png) [@erans](https://discourse.julialang.org/u/erans)\
**Post date:** [February 16, 2021, 8:49pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/14 "2021-02-16T20:49:47Z")

</div>

Indeed.

I’m actually leaning a bit more towards simply `] activate <path to my project>` which should still make `using MyPkg` possible, no?

This seems to be a bit simpler (at least for more of an “application” use-case) than the `dev` option. But I’m happy to be explained otherwise 🙂

---

<div class="post-metadata">

**Author:** ![jebej](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jebej/32/1784_2.png) [@jebej](https://discourse.julialang.org/u/jebej)\
**Post date:** [February 16, 2021, 8:57pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/15 "2021-02-16T20:57:06Z")

</div>

Yes, if you only need to use that particular package within its own environment, then simply activating that environment works!

I like to have a bunch of the packages I work on be available at the command line when I need to use them for quick commands, without having to manually activate an environment.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [February 16, 2021, 11:54pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/16 "2021-02-16T23:54:23Z")

</div>

> [@erans](#):
>
> I’m actually leaning a bit more towards simply `] activate <path to my project>` which should still make `using MyPkg` possible, no?

If you go that route, you might prefer creating a simple shell script to launch julia using the `--project /path/to/my/project` argument.

You can also check out my module “ConventionalApp.jl”:  
→ [[ANN] ConventionalApp.jl ](https://discourse.julialang.org/t/ann-conventionalapp-jl/52390)

You don’t HAVE to use ConventionalApp.jl. It is basically just a utility module to help you create bash files, etc. But it gives you _ **a** _ solution on how conventional applications can be “generated”/launched in a somewhat practical manner.

Note that ConventionalApp.jl also cleans up the `LOAD_PATH` (removes unnecessary `"@v#.#"` “project” from the environment stack).

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [February 16, 2021, 11:59pm UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/17 "2021-02-16T23:59:27Z")

</div>

> [@erans](#):
>
> Thanks for all the tips, regarding this specific point though, my main interest _currently_ is the best way to structure code. My thought was, that the “best” way would likely best leverage Julia’s capabilities. As far as I can tell, the two main Julia highlights are JIT and multiple dispatch. With that in mind, I was/am trying to figure out how my code structure can best play nice with these concepts.

If that’s the case, I suggest you take a look at my post here (and those it links to):

> [@How to load a module from the current directory?](https://discourse.julialang.org/t/how-to-load-a-module-from-the-current-directory/21727/6):
>
> I don’t think this particular behaviour is documented yet. If it is, it wasn’t visible enough for me to find it: I as well struggled with this issue a few months ago. The confusion In particular, I had to look through Julia’s code to understand that any entry in LOAD\_PATH, whose directory contains a Project.toml file, will be treated as a “project environment” only. In other words: having a Project.toml file inhibits Julia from using it as a “package repository”. An introduction, for now At …

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [February 17, 2021, 12:14am UTC](https://discourse.julialang.org/t/single-module-vs-submodules-in-a-project/55361/18 "2021-02-17T00:14:49Z")

</div>

> [@MA\_Laforge](#):
>
> creating a simple shell script to launch julia using the `--project /path/to/my/project` argument.

And if one _is_ at the project directory, it suffices do start Julia with

```julia
julia --project

```

Since it is somewhat natural to navigate to the working directory to code, that is quite practical.