# Implicitly loaded modules in the future?

**URL:** <https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646>\
**Category:** Internals & Design\
**Tags:** question, module, code-organization\
**Created:** [June 9, 2021, 8:58pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646 "2021-06-09T20:58:00Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [June 9, 2021, 11:36pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/21 "2021-06-09T23:36:37Z")

</div>

> [@panos.asproulis](#):
>
> that the proper way to arrange a code when these packages are only meaningful as parts of a single and specific application?

I would argue if it’s a big enough application, then it is. For example, whatever you would put in separate modules might make sense to be separate packages. Especially any kind of separate components of the single application. In particular, I don’t think it’s important that a package has more than one “user” (e.g. a downstream module/package).

Note that the difference betwen packages and modules is pretty small, and in particular separate packages do not have to exist in different git repositories, they do not have to be hosted online, and they do not have to registered in General. The difference is that they aren’t nested (but you can have submodules within a package, and you can extend the functions of one package via methods in another) and need separate Project.toml’s, and if you want to make use of versioning and compat, you need to register them (in your own registry). I think one way to see packages is as independently versioned modules for which there are mechanisms to express dependency relationships and compatability constraints in the stdlib Pkg. In my opinion, they are a provide a great way to have a large scalable codebase (by which I mean able to scale to allow many developers or many components, or a lot of code-- not just scalable to have many users).

I elaborate on some of the tooling and misconceptions about registries and packages in my answer [Combining two co-dependent modules into a single program - #18 by ericphanson](https://discourse.julialang.org/t/combining-two-co-dependent-modules-into-a-single-program/61881/18).

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [June 9, 2021, 11:52pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/22 "2021-06-09T23:52:58Z")

</div>

Packages can be very very small and very specific. No reason not to make a package!

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 9, 2021, 11:55pm UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/23 "2021-06-09T23:55:29Z")

</div>

Indeed, I second the above. Effectively, I have one complicated package to compute something in my field, and recently I have split one part of it into another very small package that does a part of the necessary computations of the big package. The new one has already attracted more interest than the original big package.

Related to this, what I would like to see is a more permissive way to register package names in the general registry, specifically meaning allowing names that carry the github username along with it (such as `myuser/MyPackage`). That would foment the splitting of packages into smaller units without one feeling that one is “taking” a name that could better be used by someone else.

For example, I want to have a Gram-Schmidt orthogonalization implemented for a small use case, but I don’t want to carry all the dependencies of serious implementations of that (i. e. BLAS). I would like to register my `user/GramSchmidt` and have it as a dependency of my packages, but I would not dare to register a package with that name.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 12:10am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/24 "2021-06-10T00:10:03Z")

</div>

If the package looks like it could be of interest to someone other than you, it is fine to register it, even when very small. If it is of interest only to you, no need to register it at all!

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 10, 2021, 12:44am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/25 "2021-06-10T00:44:28Z")

</div>

Quoting myself from [this response](https://discourse.julialang.org/t/ann-patmodules-jl-a-better-module-system-for-julia/52226/15) in the above linked thread regarding PatModules.jl:

> One of the reasons that Julia doesn’t use namespaces as aggressively as Python is because we group related functionality into generic functions. Instead of having `List.map` , `String.map` , `Tuple.map` , etc, we just have one generic function `map` . The emphasis is on overloading generic functions rather than putting slightly different versions of functions in separate modules. In order to fully take advantage of multiple dispatch and function overloading, you want a pretty flat namespace.
> 
> (The `List.map` , `String.map` , `Tuple.map` example is taken from languages like Erlang and Elm.)

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 10, 2021, 12:55am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/26 "2021-06-10T00:55:27Z")

</div>

> [@PetrKryslUCSD](#):
>
> If it is of interest only to you, no need to register it at all!

But how to make it a dependency of one’s packages then? (one that will be installed automatically once my _registered_ package is installed).

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 1:15am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/27 "2021-06-10T01:15:33Z")

</div>

Perhaps I misunderstand the question, but isn’t that the job of the project? Simply add the package with the github url.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 10, 2021, 1:21am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/28 "2021-06-10T01:21:52Z")

</div>

I never tried that. If I have a package registered in the general registry, can I make it depend on a package that is not in the general registry?

Other users of the registered package will have the non-registered package installed automatically when the registered package is installed?

How will be versions handled?\*

(Now that brings me another question: what happens if someone deletes a GitHub repository of a registered package? Do the registered versions continue to be available?)

PS. I not considering the possibility of having a custom registry and telling the users of my package to add it. I don’t think this is a viable alternative for most users, custom registries soon enough become too messy for the final user.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [June 10, 2021, 1:25am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/29 "2021-06-10T01:25:05Z")

</div>

> [@lmiq](#):
>
> I have a package registered in the general registry, can I make it depend on a package that is not in the general registry?

Nope

> [@lmiq](#):
>
> what happens if someone deletes a GitHub repository of a registered package? Do the registered versions continue to be available?

They won’t be accessible by git, but the Julia package servers should store any versions they’ve seen forever, so hopefully everything will still work.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 1:30am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/30 "2021-06-10T01:30:43Z")

</div>

Oh, I see what you are saying. Having a registered package dependent on something that is not registered (or even private) seems weird.

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [June 10, 2021, 3:04am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/31 "2021-06-10T03:04:50Z")

</div>

Here’s a concrete example of a problem which roughly resonates (at least in my ears) with the gripes of @panos.asproulis:

I am currently working on a multigrid solver which internally uses the conjugate residual method (an alternative implementation of MinRes) as a smoother. In an ideal world, I would like to split this code into three modules, namely one each for the multigrid and conjugate residual codes, and an additional `AbstractIterativeSolvers` module which introduces the abstract interface shared by the multigrid and conjugate residual solvers. Obviously, the dependencies between these modules are as follows:

- `AbstractIterativeSolvers` depends on nothing.
- `ConjugateResiduals` depends on `AbstractIterativeSolvers`
- `Multigrid` depends on `ConjugateResiduals` and `AbstractIterativeSolvers`

The problem then is that as I am developing these modules, I want to be able to load each of these modules and all their dependencies, but also only their dependencies. So for example, when I am testing `ConjugateResiduals`, I want to be able to load only `ConjugateResiduals` and `AbstractIterativeSolvers` and not `Multigrid` because `Multigrid` might incur a parsing error due to some unfinished edits (research software development is messy…). The current module system does not allow me to express this relationship cleanly: if I put a `include("AbstractIterativeSolvers.jl")` in both the `ConjugateResiduals.jl` and `Multigrid.jl` files, then the two modules end up using two _different_ copies of `AbstractIterativeSolvers` which defeats the point of introducing an abstract interface. On the other hand, if I don’t put these includes, then I must be careful to always reload exactly the right files depending on the changes I made.

Clearly, none of this is a problem if you are happy to either provide only a single, monolithic `Multigrid` module which takes care of loading all the dependencies in the right order, or to split `AbstractIterativeSolvers` and `ConjugateResiduals` into separate packages. However, both approaches involve a significant amount of overhead and complexity if all you want is to prototype your ideas on a timescale which is somewhere in between “I can hack this together in five minutes” and “I want this piece of code to last forever and I am willing to do whatever it takes to achieve that”.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 3:50am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/32 "2021-06-10T03:50:20Z")

</div>

I think including a source file multiple times is a not-great idea. Include once, and “use” I would say. Would that work for you?

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [June 10, 2021, 3:52am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/33 "2021-06-10T03:52:34Z")

</div>

Yes, that is of course the solution that Julia forces upon me. The point is that this is not a particularly great solution because it raises the question of where to place that single `include()`.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 3:53am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/34 "2021-06-10T03:53:38Z")

</div>

When developing, anywhere. When done developing (in rough outlines, at least), the top source file?

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [June 10, 2021, 3:57am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/35 "2021-06-10T03:57:59Z")

</div>

But why do I have to distinguish between development and production? These things always go in a cycle, so I will have to keep switching my `include`s around which is a nuisance and a source for errors.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 10, 2021, 3:58am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/36 "2021-06-10T03:58:40Z")

</div>

Then keep it in the top package source. I do. E.g. [MeshCore.jl/MeshCore.jl at master · PetrKryslUCSD/MeshCore.jl · GitHub](https://github.com/PetrKryslUCSD/MeshCore.jl/blob/master/src/MeshCore.jl)  
Edit: better example (abstract + concrete) in [https://github.com/PetrKryslUCSD/FinEtoolsDeforNonlinear.jl/blob/master/src/FinEtoolsDeforNonlinear.jl](https://github.com/PetrKryslUCSD/FinEtoolsDeforNonlinear.jl/blob/master/src/FinEtoolsDeforNonlinear.jl)

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [June 10, 2021, 5:10am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/37 "2021-06-10T05:10:57Z")

</div>

More abstractly speaking, the point is that code often has the structure of a directed acyclic graph, and the Julia module / `include()` system does not allow me to reflect that structure. I believe keeping that structure around would be useful throughout the life cycle of a piece of software, and putting all the includes in the “entry file” of a package does not do that.

Of course, this doesn’t mean that the current system is unusable. I’m just expressing my believe that there might be some room for improvement (though I have no idea what that improvement might look like).

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [June 10, 2021, 6:42am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/38 "2021-06-10T06:42:09Z")

</div>

I also think `include` is not a great way to organize code and one of Julia’s weak spots. I compared it to Go in [another](https://discourse.julialang.org/t/what-dont-you-like-about-julia-for-serious-work/54591/41) thread:

> In Go, you can’t include files: you can import packages and one package = one directory and _each file_ has an explicit list of imported packages at the top, and importing names in the main namespace (like `using` in Julia) is strongly discouraged. In Julia, files can include files can include files… Open a file, you don’t know in which context it is included so you can’t be sure how it’s going to be interpreted. Maybe it’s even included several times in different contexts? Maybe the behavior depends on the order of includes?
> 
> Another way to put it: declarative (rather than imperative) code is generally considered more robust or at least easier to analyze and verify. In Go, you write imperative code in functions, but the overall organization (definition of packages and functions) is declarative. In Julia, this organization itself is imperative.

Note that Jeff voiced some [agreement](https://discourse.julialang.org/t/what-dont-you-like-about-julia-for-serious-work/54591/63) 🙂

---

<div class="post-metadata">

**Author:** ![patrick-kidger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrick-kidger/32/20378_2.png) [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)\
**Post date:** [June 10, 2021, 10:57am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/39 "2021-06-10T10:57:30Z")

</div>

> [@ettersi](#):
>
> More abstractly speaking, the point is that code often has the structure of a directed acyclic graph, and the Julia module / `include()` system does not allow me to reflect that structure.

^ I completely agree, and this is the key point IMO.

> Open a file, you don’t know in which context it is included so you can’t be sure how it’s going to be interpreted. Maybe it’s even included several times in different contexts? Maybe the behavior depends on the order of includes?

And problems like this are the result.

It’s actually pretty straightforward to demonstrate genuine spooky-action-at-a-distance as a result of the typical `include()` pattern.

I’ll reiterate the mention of [`FromFile.jl`](https://github.com/Roger-luo/FromFile.jl) from earlier in the thread – this package introduces a custom import system that fixes all these issues. Never write an `include()` again.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 10, 2021, 11:02am UTC](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646/40 "2021-06-10T11:02:49Z")

</div>

What if modules behaved as Pluto notebooks? (what you see is what you get). That could be even the place for a macro.

[Previous page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=1)

[Next page](https://discourse.julialang.org/t/implicitly-loaded-modules-in-the-future/62646.md?page=3)