# Implicitly loaded modules in the future?

**URL:** <https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646>\
**Category:** Internals & Design\
**Tags:** question, module, code-organization\
**Created:** [June 9, 2021, 8:58pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646 "2021-06-09T20:58:00Z")\
**Posts on this page:** 20\
**Page:** 14

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 24, 2021, 12:31am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/264 "2021-06-24T00:31:54Z")

</div>

> [@jzr](#):
>
> On (2), I don’t get what _non-determinism_ refers to in these examples.
> 
> On (4), I agree that when it’s hard “to know the actual structure of the code because it’s not written but generated” that’s a downside, but I would have described the current situation that way, not the proposed one. Currently when reading `B.jl` , I don’t know what `common` refers to without looking at the context in which `B.jl` is included. In the proposals, it is clear what `common` refers to. So I don’t get how the proposal makes the structure less clear in your perspective. I would like to understand that.

Just see the earlier discussion:

> [@Implicitly loaded modules in the future?](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/56):
>
> That’s nice, but that does mean the final ordering in which the code is actually loaded is implicit. So I agree “moreover anything implicit is just an easy source of bugs, really”, which is the worrisome issue for having it all worked out implicitly by a resolver on a DAG. include forces you to choose an order. That orders makes all things well-defined: that order makes you able to know what any values of overridden globals would be. The other method can mean if you do Main.a = i in modules A\_…

> [@Implicitly loaded modules in the future?](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/61):
>
> This can show up in many ways though. If you have two modules which overload the same Base function, but with different methods on the same dispatch, the final method that you get is now dependent on the DAG. If you want to hash all of the imported code to say, track code changes to remove precompilation, you can have invalidation occur because the DAG’s heuristics change. One major global that is hit in all languages is I/O: if two modules write to a file at load time, the ordering of the cont…

> [@Implicitly loaded modules in the future?](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/62):
>
> Oh yeah, that one was patched out of the language to be illegal. But on refs it still works: x = Ref(2) module A Main.x[] = 1 end using .A @show x This is actually a nice way to debug FWIW, because you can grab pieces of a big function to play with later. Then of course you can do this with arrays, files, etc.

The most insidious is that it would effect I/O at loading.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 24, 2021, 4:40am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/265 "2021-06-24T04:40:07Z")

</div>

I could get behind @Skoffer’s idea regarding the keyword `depends`. I would favor the word `depends` over `import`, since `import` is already used to mean that you are importing objects from a module.

@patrick-kidger My apologies if the `depends` idea is essentially what you had in PatModules.jl… perhaps I didn’t look closely enough at the details of that package.

The advantages of `depends`, from my point of view:

- I no longer have to manually order my `include` statements.
- It does not enforce a file-module correspondence.
  - Thus, I can still use the flat namespace that I prefer.

However, the example that @Skoffer provided is not the best example, because you can define your methods of `foo` in whatever order you want. There’s no sense in which the `A.jl` and `B.jl` files depend on the `utils.jl` file. (`function foo end` is not more primal than any other method.) If we ignore modules that run imperative code, the only thing that matters is that types are defined before they are referred to.

Let me elaborate a bit more on why I don’t like turning every file into a module. Suppose I have a generic function `foo`, with various methods, that gets used throughout a package `DemoPackage.jl`. With Julia as it is now, `foo` “belongs” to the `DemoPackage` module. But if every file now has to be a module, I have to arbitrarily pick a submodule to “own” `foo`, and then I have to import `foo` from my arbitrarily chosen submodule every time that I want to extend it.

Below is a runnable example of what this would look like, using the current module system with nested modules. I think it would look basically the same if the nested modules in the example were separate files with the file-module correspondence enforced.

```julia
module DemoPackage
       
	module A
		struct S end
		foo(::S) = 1
	end
	
	module B
		import ..A: foo
		struct T end
		foo(::T) = 2
	end
	
	module C
		import ..A: foo
		struct U end
		foo(::U) = 3
	end
	
	import .A: S
	import .B: T
	import .C: foo, U
	export foo, S, T, U
	
end

```

```julia
julia> using .DemoPackage

julia> methods(foo)
# 3 methods for generic function "foo":
[1] foo(::S) in Main.DemoPackage.A at REPL[1]:8
[2] foo(::T) in Main.DemoPackage.B at REPL[1]:14
[3] foo(::U) in Main.DemoPackage.C at REPL[1]:20

```

Note how I arbitrarily picked module `A` to “own” `foo`. But in reality, none of the modules really owns `foo`—it is a generic function who’s full definition spans multiple modules. There are other arbitrary choices I had to make here:

- In module `C`, I could have done `import ..B: foo` instead of `import ..A: foo`, because, now `B` “owns” `foo` just as much as `A` does!
- In the third to last line, I could have done either `import ..A: foo`, `import ..B: foo`, or `import ..C: foo`. It’s an arbitrary choice, because they all refer to the _same generic function_.

To make matters worse, I’ve actually introduced an artificial code dependency that wouldn’t have existed otherwise. Look what happens if I transpose the definition of module `B` above the definition of module `A`:

> **DemoPackage module with order of A and B flipped**
>
> ```julia
> module DemoPackage
>        
> module B
> import ..A: foo
> struct T end
> foo(::T) = 2
> end	
> 
> module A
> struct S end
> foo(::S) = 1
> end
> 	
> module C
> import ..A: foo
> struct U end
> foo(::U) = 3
> end
> 	
> import .A: S
> import .B: T
> import .C: foo, U
> export foo, S, T, U
> 	
> end
> 
> ```

If I run the new `DemoPackage` with the order of `A` and `B` flipped, I get `ERROR: UndefVarError: A not defined`. You might say, “That’s exactly how it’s supposed to work. The code dependency has been enforced.” But the point is, there should not be a code dependency here, because I can define methods in any order I want! I could have done the following, where I can flip the order of defining `foo(::S)`, `foo(::T)`, and `foo(::U)` to my heart’s content:

```julia
module DemoPackage

    struct U end
    struct T end
    struct S end

    foo(::U) = 3
    foo(::T) = 2
    foo(::S) = 1

    export foo, S, T, U

end

```

To summarize:

- Using the same generic function across multiple submodules requires arbitrary import choices and introduces artificial and unnecessary code dependencies.
- You can’t really encapsulate a generic function inside a module, because the generic function is defined by its _global_ method table.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 24, 2021, 5:45am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/266 "2021-06-24T05:45:35Z")

</div>

Here’s another example that demonstrates the vacuity of a system like `FromFile.jl`. Suppose we start with this:

```julia
# A.jl
function foo end

# B.jl
@from "A.jl" import foo
bar(::Int) = 1
foo(x) = bar(x) 

```

Ok, great, now we know that file `B.jl` only depends on file `A.jl`. But suppose we add another file like this:

```julia
# C.jl
@from "B.jl" import bar
bar(::Float64) = 2 

```

The new package with the addition of the `C.jl` file will load and run just fine. But now the behavior of `foo` is different, because we’ve overloaded `bar`. File `B.jl` asserts that its contents, including `foo`, only depend on file `A.jl`, but that is wrong! In fact, the behavior of `foo` now also depends on file `C.jl`, even though there is no `@from "C.jl"` statement at the top of `B.jl`.

To summarize:

- The `@from` dependencies listed at the top of a file are _incomplete_.
- Modules do not isolate dependencies in a language where every function has a global method table.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [June 24, 2021, 6:54am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/267 "2021-06-24T06:54:20Z")

</div>

> [@CameronBieganek](#):
>
> Note how I arbitrarily picked module `A` to “own” `foo` . But in reality, none of the modules really owns `foo` —it is a generic function who’s full definition spans multiple modules. There are other arbitrary choices I had to make here:
> 
> - In module `C` , I could have done `import ..B: foo` instead of `import ..A: foo` , because, (shock!), now `B` “owns” `foo` just as much as `A` does!
> - In the third to last line, I could have done either `import ..A: foo` , `import ..B: foo` , or `import ..C: foo` ! It’s an arbitrary choice, because they all refer to the _same generic function_ !

How does ownership of the function mesh with what Julia reports as parent module for `foo`?

```julia
julia> parentmodule(foo)
Main.A

julia> parentmodule(B.foo)
Main.A

```

Even though `foo` gets imported in B and C isn’t that merely bringing in a function from a different module as an alias in the current module? Sure, each module owns a method of `foo`, but I would actually say that neither B or C own the function (although they clearly influence its definition).

Edit: ah, just noticed that `parentmodule()` can take a second parameter for specifying a particular type signature for a function. When left off it reports the first definition of that function.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 24, 2021, 7:17am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/268 "2021-06-24T07:17:52Z")

</div>

> [@patrick-kidger](#):
>
> Some folks seem to prefer the current system simply because it makes it very difficult to maintain any internal structure

I think you misunderstand. I can’t speak for others, but

1. I don’t find organizing the internal structure of my packages difficult at all (and I use submodules when I need to),
2. I like the current system because it is simple, and decouples `using` etc from the file system. I consider this an elegant feature, which would be nice to keep.

> [@patrick-kidger](#):
>
> no-one seems to be advocating monorepos […] I don’t see that this is relevant to the discussion.

It is implicitly assumed that difficulty of internal organization of packages increases with the amount of code in a package.

IMO maintaining complex and large packages is key to this discussion. After all, for a small package everything can go in one file, or maybe a few files with a couple of `include`s, and the issue becomes less pressing. It is no coincidence that a lot of package authors have a difficult time understanding the motivation for the proposal.

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [June 24, 2021, 8:58am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/269 "2021-06-24T08:58:04Z")

</div>

> [@CameronBieganek](#):
>
> Note how I arbitrarily picked module `A` to “own” `foo` . But in reality, none of the modules really owns `foo` —it is a generic function who’s full definition spans multiple modules. There are other arbitrary choices I had to make here:
> 
> - In module `C` , I could have done `import ..B: foo` instead of `import ..A: foo` , because, (shock!), now `B` “owns” `foo` just as much as `A` does!
> - In the third to last line, I could have done either `import ..A: foo` , `import ..B: foo` , or `import ..C: foo` ! It’s an arbitrary choice, because they all refer to the _same generic function_ !

So @Skoffer raised a similar point before. I agree this seems like a weakness. I would note that the current approach doesn’t really change this though. It just hid it from you, by “implicitly importing” everything when you use `include`. (So to speak.)

FWIW, in practice I find most of my uses of multiple dispatch do involve a natural “owner” of the function, whilst the others are merely seen to be extending it with additional methods.

> [@CameronBieganek](#):
>
> To make matters worse, I’ve actually introduced an artificial code dependency that wouldn’t have existed otherwise. Look what happens if I transpose the definition of module `B` above the definition of module `A` :
> 
> DemoPackage module with order of A and B flipped

This at least shouldn’t be an issue. Part of the current discussion is that this kind of topological sorting no longer becomes something needing manual resolution: the statement for import `A` will trigger the loading of `A`.

> [@CameronBieganek](#):
>
> Here’s another example that demonstrates the vacuity of a system like `FromFile.jl` .

Yep, I’m aware of this example. I would respond by noting that:

- IMO it’s still a big improvement on the current system. Now you know you depend on the _function_ `foo` (if not which method). This is more than you could say before.
- One big win of this proposal is that it stops your problem scenario _from happening accidentally_. It’s very easy to have some kind of

```julia
# main.jl
include("A.jl")
include("B.jl")
...
include("Z.jl")

# A.jl
def foobar(...) = ...

# Z.jl
def foobar(...) = ...

```

in which `A.jl` creates a function with a very generic name (`foobar`), and then `Z.jl`, instead of creating a new function, actually creates a new method. Potentially changing the behaviour in the almost completely unrelated `A.jl`. Very easy to imagine happening in larger projects.

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [June 24, 2021, 9:06am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/270 "2021-06-24T09:06:54Z")

</div>

> [@Tamas\_Papp](#):
>
> IMO maintaining complex and large packages is key to this discussion

The key point is that – to use SciML as an example – it makes sense to split things up into multiple packages. The components are large and decoupled enough that this is sensible. So “what if we made SciML a monorepo” isn’t necessarily a good example.

Probably the best examples would be commerical software products (based on my experience with other languages). Projects which are large, complex, and which your latest junior dev can contribute to, whilst only needing to understand a subset of the codebase. And which the language largely isolates from making changes that affect the rest of the codebase. My interest in this feature is really about making this kind of commercial-scale stuff easier to manage.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [June 24, 2021, 9:30am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/271 "2021-06-24T09:30:09Z")

</div>

I haven’t read the full series of posts, but for a large, complex codebase that went the monorepo route:

> **[MathOptInterface.jl/src at master · jump-dev/MathOptInterface.jl](https://github.com/jump-dev/MathOptInterface.jl/tree/master/src)**
>
> master/src

It uses a lot of nested submodules that can import from each other via `using ..`.

For example, the `MathOptInterface.FileFormats.MPS` sub(sub)module can use things defined in `MathOptInterface.Utilities` submodule:

> <https://github.com/jump-dev/MathOptInterface.jl/blob/9fb8da1c8a12a1cac74088c3192175e6c982733e/src/FileFormats/MPS/MPS.jl#L3-L6>

If you want to be sharing some `utils.jl` file between other files, make it a submodule and import that.

Maybe I need to read the full series of posts…

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 24, 2021, 9:35am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/272 "2021-06-24T09:35:27Z")

</div>

> [@patrick-kidger](#):
>
> So “what if we made SciML a monorepo” isn’t necessarily a good example.

Sure, but that’s not the question here. The relevant point is that SciML, JuMP, JuliaImages, JuliaSmoothOptimizers and many similar projects are already large and complex enough that their principal architects are in a good position to speak about the benefits of a proposal like this one from their own perspective, and it would be interesting to hear their opinion.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 24, 2021, 10:34am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/273 "2021-06-24T10:34:29Z")

</div>

> [@patrick-kidger](#):
>
> Probably the best examples would be commerical software products (based on my experience with other languages). Projects which are large, complex, and which your latest junior dev can contribute to, whilst only needing to understand a subset of the codebase. And which the language largely isolates from making changes that affect the rest of the codebase. My interest in this feature is really about making this kind of commercial-scale stuff easier to manage.

Sure we can move the goal post, but [Pumas](https://pumas.ai/) is a great example for this. It’s a already quite a “commerical” success (commercial in America California?): it is already one of the FDA eCTD standards for clinical trials, and remember “Pumas has emerged as our “go-to” tool for most of our analyses in recent months” by Husain, the head of clinical pharmacology at Moderna in 2020. Similarly includes work with [Pfizer QSP team](https://juliacomputing.com/case-studies/pfizer/), [United Therapeutics](https://juliacomputing.com/case-studies/united-therapeutics/), etc., so full enterprise and something that’s really out there now with a relatively good sized team of full time engineers.

It’s built the same way as SciML with separate repos for separate functionality, though there is a slightly larger core in Pumas.jl. It’s interesting to see why it’s done this way though. In the Pumas case, it’s pretty necessary because the alternative repos are related to differently licensed products, such as Lyv, OptimalDesign, or the coming [Pumas-QSP](https://youtu.be/KLjJps268Wc). So the split is not just for development reasons, but also for licensing and access rights. It also helps with the marketing as well, because then specific areas can be branded, talked about in workshops, and focused in videos with clear naming schemes.

[JuliaSim](https://juliacomputing.com/products/juliasim/) is the new modeling and simulation product coming out of Julia Computing, with [a quick overview here](https://www.youtube.com/watch?v=oXcyPS93L8c) and a deeper technical discussion [to occur at the 2021 Modelica Conference](https://arxiv.org/pdf/2105.05946.pdf) and at JuliaCon 2021 (register everyone!). It again has this same development split, this time not because of licensing (because instead the coming model is a cloud license to the full suite), but because there is a major separation of concerns between HVAC models and electrical circuits.

If you look at similar simulation-based products in the area, such as [Modelon’s model libraries](https://www.modelon.com/products-services/modelon-library-suite-modelica-libraries/) and [Ansys products](https://www.ansys.com/products#t=ProductsTab&sort=relevancy&layout=card), they similarly have this form of modularization, this time for licensing reasons again, but also for marketing. If you look at the marketing materials that are online, they are per module and almost certainly just pick up the portions relating to a given industry/customer quickly piece together new demonstrations. Many times you can then have the sales team divide into specific specialties, which makes sense as they won’t have the technical knowledge behind a lot of the work so focusing on one area will allow a bit more depth (and building a better network).

Thus while I know Google famously uses a monorepo, FAANG companies are rather outliers and recognizing how and why they are different is important to business success. Monorepos do have the advantage of immediately notifying downstream effects of a PR, and when people are employees you’re much more able to say “you have to fix that by Friday, pull in X from the deployment team if you don’t know how”. And for a purely cloud-based product where you don’t have to license things separately, it does make the deployment of a single sign-on much easier. But then again, it does add developer burden, which Google is able to handle by giving famously high salaries and hiring some of the best engineers there are to offer, so if your PR in ML packages breaks an RSS feed in the webserver, you’re supposedly smart enough to be able to cross domains and fix it.

Google also writes a large fraction of its own dependencies, and so not requiring a modular system can make sense. Counter that to these Julia-based organizations which require a form of modularity from the start because they are already based on open source software (such as the SciML organization), and so even if there was a monorepo there would still be dependency management required. That said, monorepos do seem pretty enticing for cloud products if you’re mostly focusing on the deployment side.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [June 24, 2021, 12:30pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/274 "2021-06-24T12:30:32Z")

</div>

> [@patrick-kidger](#):
>
> Probably the best examples would be commerical software products (based on my experience with other languages). Projects which are large, complex, and which your latest junior dev can contribute to, whilst only needing to understand a subset of the codebase. And which the language largely isolates from making changes that affect the rest of the codebase. My interest in this feature is really about making this kind of commercial-scale stuff easier to manage.

I’m also interested in these questions from an industry point of view. I won’t claim that we have a huge amount of Julia code, nor are we a large company, but we have enough code that scaling issues need to be considered.

In my experience include order is a small irritation. Not a non-issue but also not a significant one. I have never once seen a double include issue and in fact have trouble understanding why people are even talking about it (no need to rehash that, it’s there somewhere in this thread). I have some sympathy for the viewpoint that it should be easy to predict or find where a function is defined but not to the point that I want to see a forced mapping to filenames.

To some extent this can be influenced by the fact that I’ve found submodules to be rather unergonomic and much prefer to split code into packages. However, even if Julia packages are rather lightweight, it doesn’t scale to split all your code into a large number of micro-packages. There’s a bit too much ceremony around creating a package, the registration overhead is not negligible (even more so in General than in a private registry), and it’s a mental burden to keep track of hundreds of small packages and their versions and compat. With subdir packages you can avoid also having hundreds of small repositories, but that’s only a minor gain.

For a concrete example we have an internal file format implemented in a Julia package. This consists of a write function and a read function. The writer is slow, requires a lot of dependencies and is basically run once for a given set of data, offline. The reader is fast, used online and needs no dependencies. The fact that users of the reader pull in lots of irrelevant dependencies could be solved by splitting out the reader into its own package but it’s not really attractive. The writer and reader are inherently coupled, need to be tested together and if the file format is changed in some way they need to be updated in tandem.

What I would like is to have the power of packages available at the submodule granularity, with less overhead than full packages. I.e. a subpackage concept that could have the following characteristics:

- Subpackages relate to the _directory_ structure of a package. When  
you `import P.A.B` Julia will load the file `<path to P>/src/A/B/B.jl`  
and `import` the module `B`, which must be defined in the file. Loading of subpackages should not require the full package to be loaded as well.

- Subpackages share UUID, version number, and `Project.toml` with the  
main package.

- Subpackages can have different sets of dependencies and other  
packages can depend on specific subpackages. `compat` is shared with  
the main package and can only be specified at the main package  
level.

- Only the main package is registered and subpackage information is  
stored in the registry files for the main package.

And yes, I’m aware that this might be breaking and/or fundamentally difficult to implement. There are many details that would need to be worked out and tested. But I would find it a lot more useful than other ideas that have been discussed in this thread.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 24, 2021, 12:40pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/275 "2021-06-24T12:40:45Z")

</div>

> [@GunnarFarneback](#):
>
> What I would like is to have the power of packages available at the submodule granularity, with less overhead than full packages. I.e. a subpackage concept that could have the following characteristics:

The current subpackage infrustructure is close. JuliaImages used to do this but went back IIRC. KernelAbstractions.jl is a good example that uses this for the CUDAKernels and ROCKernels subpackages, which [live in the same repo](https://github.com/JuliaGPU/KernelAbstractions.jl/tree/master/lib) but are [different registered packages](https://juliahub.com/ui/Packages/CUDAKernels/fgh4c/0.2.1) with different dependencies, so it has a separate Project.toml. Indeed I think this structure could be improved with shared Project tomls when dependencies are the same, though I wonder what kinds of extra complexity having such a system in the package manager could cause.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [June 24, 2021, 1:21pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/276 "2021-06-24T13:21:57Z")

</div>

> [@ChrisRackauckas](#):
>
> The current subpackage infrustructure is close.

Not really. The usual terminology is subdir packages and there’s nothing “subpackage” about them. They are fully independent packages which just happen to be co-located within a common repository.

(The second use of subdir packages is when you have one Julia package which only is a part of a repository, e.g. a Julia wrapper package for a C library in the same repository. This use case works great and was the reason I helped push subdir packages over the finish line for Julia 1.5.)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 24, 2021, 1:28pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/277 "2021-06-24T13:28:58Z")

</div>

> [@GunnarFarneback](#):
>
> Not really. The usual terminology is subdir packages and there’s nothing “subpackage” about them. They are fully independent packages which just happen to be co-located within a common repository.

I just mean close in terms of implementation distances. To go from this to something that can opt-out of having different dependencies and ship with the main package doesn’t seem to far off, though it could add some odd complexity.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 24, 2021, 1:31pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/278 "2021-06-24T13:31:22Z")

</div>

> [@patrick-kidger](#):
>
> This at least shouldn’t be an issue. Part of the current discussion is that this kind of topological sorting no longer becomes something needing manual resolution: the statement for import `A` will trigger the loading of `A` .

Yes, I understand that. The point was that the dependency is _artificial_. The methods of a generic function can be defined in any order, so `B` doesn’t really depend on `A`, it’s just forced to import `foo` from `A` to ensure that `B` overloads the same generic `foo`.

> [@patrick-kidger](#):
>
> IMO it’s still a big improvement on the current system. Now you know you depend on the _function_ `foo` (if not which method). This is more than you could say before.

If I’m reading a file and I see the line `baz(x) = foo(x)`, now I know that this file depends on the function `foo`. 😅 And because all functions are generic with global method tables, the actual behavior of `foo` depends on any number of `foo` method definitions which can occur anywhere before or anywhere after the line `baz(x) = foo(x)` (even in a different module!). An import statement like `from A import foo` implies that everything I need to know about `foo` can be found in `A`, but it’s just not true—not even close to being true.

(Luckily the mental burden of understanding `foo(x)` is not that bad, because generic functions are given generic docstring definitions that are intended to apply to each method. In other words, `foo(x)` has approximately the same meaning regardless of the data type of the input.)

> [@patrick-kidger](#):
>
> I agree this seems like a weakness. I would note that the current approach doesn’t really change this though. It just hid it from you, by “implicitly importing” everything when you use `include` .

If I have all my `foo` definitions in the same, flat, package namespace, I don’t think there’s any “implicit importing” going on. It’s exactly the opposite, in fact. It’s all the same namespace, so of course I don’t have to do any importing (implicit or not).

---

<div class="post-metadata">

**Author:** ![mhinsch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mhinsch/32/26676_2.png) [@mhinsch](https://discourse.julialang.org/u/mhinsch)\
**Post date:** [June 30, 2021, 11:43am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/279 "2021-06-30T11:43:21Z")

</div>

I want to add a slightly different perspective. I think solving this issue can also be beneficial for small projects. I myself work on agent-based models of social systems with code sizes that are minuscule compared to many of the projects mentioned here.

However, it would still be nice to be able to split my code into independent units without having to jump through hoops. As an example:

In a given project I might - besides the main model - have some code that generates an artificial geography, code responsible for setup, a belief dynamics sub-model, some bits that are responsible for the gui, etc. All of these parts cover well-described and separate functionality, so they should live in their own modules which I would like to keep in their own separate (local) files/directories. So far so good, if I want to use any of these modules I just do the not pretty but workable `include("M1.jl"); using M1.jl`.

The problem is, there might be cross-dependencies between these modules. For example, the belief dynamics might be used by the main model as well as the setup. If I keep doing `include`+`using` at point of use I might start running into double include issues. The canonical solution for that at the moment seems to be to have some sort of `main.jl` file that `include`’s all of the single module files and then do `using` to my heart’s content wherever necessary.

In my opinion that’s not a very satisfactory solution, however. It makes it impossible to track dependencies _locally_, i.e. at the point where they are used. If I look at my setup code for example I want to see immediately that it depends on Beliefs. I _could_ have the `using` locally but then I would have the dependency split over two completely separate places (with the `include` still living in `main.jl`)!

This becomes even more aggravating if I have different configurations of the model that use different configurations of modules. I might for example have a `run_alt.jl` that uses a different setup that does not depend on my geography module. Now I need a different `main.jl` for a different combination of module files. I find having to keep track of these dependencies globally quite unintuitive and error-prone.

My workaround so far has been to globally (i.e. somewhere at the beginning of the main script) add the current directory to `LOAD_PATH` and then just use `using` wherever necessary and/or convenient. It works, but it would be nice if the language offered a more canonical solution.

P.S.: [This](https://github.com/JuliaLang/julia/issues/4600#issuecomment-870884480) is my attempt at a solution.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [June 30, 2021, 12:39pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/280 "2021-06-30T12:39:27Z")

</div>

Can you explain “so they should live in their own modules” part? What are the benefits of putting everything in a separate modules? I mean, it is reasonable to put everything in a separate files - it helps with readability, but what is the purpose of wrapping everything in modules? The only reason, I can think of, if you have functions that have a different meaning, but overlapping signatures, then yes, modules are helping you to distinguish them. But other than that it seems that putting everything in modules wouldn’t do any good, quite the contrary, it adds artificial restrictions, lots of redundant `using/import`, and interfere with multiple dispatch.

---

<div class="post-metadata">

**Author:** ![mhinsch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mhinsch/32/26676_2.png) [@mhinsch](https://discourse.julialang.org/u/mhinsch)\
**Post date:** [July 1, 2021, 11:20am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/281 "2021-07-01T11:20:49Z")

</div>

Hmm, to be honest putting self-contained functionality into modules seems so self-evidently reasonable to me that I haven’t really thought about it explicitly so far.

I think it’s basically the same reasoning as that behind writing separate classes for separate things in an OOP language or having as few global variables as possible. You want to minimise the _potential_ of unintended interactions. That avoids bugs but it also serves as a cognitive help. When I am working on some self-contained code that is inside a module (as opposed to an “open” file) I _know_ that the only interaction it will have with the outside world is via local imports and the things the module itself exports (not strictly true of course, but if people start messing with module internals they are on their own), making it easier to understand. The other effect is that it nudges me towards increased discipline. A module is a very strong declaration of intent to keep the functionality limited to one specific thing and thus counteracts the temptation to “just put that bit there” because it’s more convenient.

Of course, as usual, none of this would be necessary if we assumed a perfect genius programmer, but then that perfect genius programmer wouldn’t need Julia but could just write in assembly. In a way most of what happened in applied computer science since the 50s or so is developing cognitive crutches for imperfect humans.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [July 1, 2021, 12:19pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/282 "2021-07-01T12:19:36Z")

</div>

I think your considerations are reasonable. But with the possibilities that Julia offers right now, a good way of thinking is:

_Your modules only exist/are relevant to your own package. Thus, it might not be a big inconvenience that your main module starts with:_

```julia
module MyPackage
  include("./mysubmodule1.jl")
  include("./mysubmodule2.jl")
  ...
end

```

where, for example, `mysubmodule2.jl` has

```julia
module MySubmodule2
  using ..MySubModule1
  ...
end

```

without repeated includes. Maybe not ideal, but since your submodules are only relevant in the context of your package, that is not too bad. (or use FromFile.jl, mentioned here many times).

If, alternatively your modules might be useful outside the context of that specific package, then make of them new packages.

---

<div class="post-metadata">

**Author:** ![mhinsch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mhinsch/32/26676_2.png) [@mhinsch](https://discourse.julialang.org/u/mhinsch)\
**Post date:** [July 1, 2021, 12:33pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/283 "2021-07-01T12:33:52Z")

</div>

Or just set `LOAD_PATH` to include the current directory, or use the macros shown [here](https://github.com/JuliaLang/julia/issues/4600#issuecomment-870884480).

[Previous page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=13)

[Next page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=15)