# Proper way of organizing code into subpackages
**URL:** https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835
**Category:** New to Julia
**Tags:** packages, code-organization
**Created:** [January 4, 2021, 5:34pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835 "2021-01-04T17:34:30Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [January 5, 2021, 9:02pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/21 "2021-01-05T21:02:52Z")
> [@patrick-kidger](#):
>
> we should already know whether or not it is a script. (And I’d note that the same is already true of the current approach when accessing a file’s contents via `include` .)
I’m afraid that is most likely not true. At the moment users are happily loading functionality from packages and modules whose internals they had never seen before. Of course, it is possible that what you propose would be enabled only for package developers, which I believe would be safe(r) than opening it up to any and all.
---
### Author:  [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)
#### Post date: [January 5, 2021, 9:17pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/22 "2021-01-05T21:17:19Z")
> [@PetrKryslUCSD](#):
>
> I’m afraid that is most likely not true. At the moment users are happily loading functionality from packages and modules whose internals they had never seen before. Of course, it is possible that what you propose would be enabled only for package developers, which I believe would be safe® than opening it up to any and all.
Fortunately, that’s not what’s being discussed.
An external package can (and should) still be `import`ed in the way that you’re familiar with. The use of `from` is specifically for relative imports within your own package.
if you’re skeptical then I’d suggest familiarising yourself with Python’s tried-and-tested import system. In a case of convergent evolution rather than explicit inspiration, the two systems are very similar.
---
### Author:  [@fengkehh](https://discourse.julialang.org/u/fengkehh)
#### Post date: [January 5, 2021, 9:26pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/23 "2021-01-05T21:26:25Z")
I think flat out preventing everyone from importing files on a whole in order to stop some from possibly doing things wrong is getting dangerously close to starving yourself in order to prevent choking hazards. If anything it probably encouraged an overuse of `include()` for those that didn’t know how to import from files properly in the first place. For those that wanted to know how to do things properly the only thing it does seem to be a whole lot of hassle or forcing them to adopt a very flat project structure. As I outlined above, a flat project structure doesn’t always make sense. In fact, one can argue that since problems in general are solved by logically breaking them down into smaller and more specific chunks, a flat project structure rarely makes sense except for fairly small projects.
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [January 5, 2021, 10:50pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/24 "2021-01-05T22:50:12Z")
I don’t think that anyone is trying to force you into anything 🙂 The opinions might sound hard but in the end people care about the succes of your project.
The main benefit, I think, of a flatter structure is that you can more easily re-use functions. In OOP, your methods are linked to your objects. So, those can be placed together in a deeply nested file. In Julia, those methods can be much more generic, so you can more often move them further up since you can use them for more objects (types).
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [January 5, 2021, 11:05pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/25 "2021-01-05T23:05:43Z")
Nothing wrong with a deeper structure of the project. My own projects are often based on modules and submodules. I’m just saying that organization based on files may be fragile. Python’s structure is also based on modules, even though superficially it might look like it is based on files.
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [January 5, 2021, 11:25pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/26 "2021-01-05T23:25:03Z")
What do you mean by fragile?
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [January 5, 2021, 11:42pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/27 "2021-01-05T23:42:17Z")
Well, it seems to me that files are more in flux than modules. Let us say I want to use `using Mod1: foo`. No matter in which file I placed this function, it will be loaded correctly. However, if I refer to a specific file in which this function supposedly resides, whenever I move the function into a different file, I have to visit all the places that load this function to change the reference.
---
### Author:  [@fengkehh](https://discourse.julialang.org/u/fengkehh)
#### Post date: [January 5, 2021, 11:52pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/28 "2021-01-05T23:52:46Z")
This I do agree (and it’s one of the criticisms I raised against Python’s filesystem based module in my first post). However doing this has its benefits, namely that it’s very clear where the code resides in and therefore very easy for the user to look up the actual implementations. As it is Julia’s module import only shows the module name. Combined with the lack of proper code tracking by ANY IDE it effectively renders nested structure useless for prototyping purpose. This is why I said this system is forcing us to adopt a flat structure - because while it is possible to nest it is not practical.
Also, I’m not sure if I agree a flat structure is inherently a better fit for multiple dispatch/data-only OO Julia uses. The reason is…well, why? Here’s a toy example I cooked up last night to test nesting: [https://github.com/fengkehh/julia\_nesting\_test](https://github.com/fengkehh/julia_nesting_test)
It’s meant to be convoluted and ridiculous because I wanted to break things by nesting. You can try running main\_test.jl and modify things in all the modules to your own liking. AFAIK I could arbitrarily import type defs and functions from any level to any level regardless where the files reside and everything still works as expected. If it wasn’t for the lack of proper code inspection support I would happily take that as a good enough workaround and use it in commercial development already.
---
### Author:  [@bert](https://discourse.julialang.org/u/bert)
#### Post date: [August 20, 2022, 1:43am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/29 "2022-08-20T01:43:06Z")
Reading this thread a year and a half later, it seems like (relative) imports are major pain point of Julia for package authors? I’m comparing to something like Rust, where you just write `use path::to::module` wherever it’s needed and boom it’s loaded, no need to worry about multiple inclusions conflicting or struggling with your IDE not understanding your code because all its imports are missing because they’re imported only in the “main” file, where the current file is included, but not in the current file itself.
I think Julia needs some sort of officially supported “import once” functionality, where, say `@include "path/to/file.jl"` or `@include "path/to/file.jl" ModuleFoo ModuleBar` that includes the file at the top level if it hasn’t already been, and if using the second form brings the specified modules into scope.
It’s just too painful as a package author to deal with imports based on C’s #include directive without even having the static guarantees of C’ compiler to help out. Is this something that could be slated for Julia 2?
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [August 20, 2022, 2:03am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/30 "2022-08-20T02:03:26Z")
I think you are misunderstanding how imports work.
---
### Author:  [@bert](https://discourse.julialang.org/u/bert)
#### Post date: [August 20, 2022, 3:06am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/31 "2022-08-20T03:06:48Z")
I understand how they work. The point is that how they work is incredibly hard to to actually be productive with. There’s a reason all languages abandoned the C-style copy-paste #include directive ages ago, which is that it’s really really hard to work with.
If you look at most widely used Julia packages, they break their source tree into several files. In most cases, the non-“main” files don’t include/using their dependencies because those dependencies are brought into scope in the “main” file. Trying to edit a file none of whose dependencies are in scope in the file itself is a major headache. Similarly, trying to chase down the origins of exports in the “main” file is a Herculean effort. You should never have to ctrl-f your entire workspace for the origin of a symbol that’s used in the current file – that’s a sign of bad design.
---
### Author:  [@liuyxpp](https://discourse.julialang.org/u/liuyxpp)
#### Post date: [August 20, 2022, 3:11am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/32 "2022-08-20T03:11:18Z")
> [@bert](#):
>
> the non-“main” files don’t include/using their dependencies because those dependencies are brought into scope in the “main” file
This is actually really painful when you try to read others’ packages. Sometimes it takes ridiculously long time to find where does a method (or any term) comes from. Is it comes from an external package imported in the main file, or is it defined else where in the current package?
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [August 20, 2022, 3:12am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/33 "2022-08-20T03:12:24Z")
> [@bert](#):
>
> There’s a reason all languages abandoned the C-style copy-paste #include directive ages ago, which is that it’s really really hard to work with.
I don’t think you do. Imports are not #includes.
---
### Author:  [@bert](https://discourse.julialang.org/u/bert)
#### Post date: [August 20, 2022, 3:15am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/34 "2022-08-20T03:15:25Z")
Correct. `include`s are `#include`s. Therein lies the problem. in order to `import` a module, it has to first be in scope. This means it either has to be a package declared as a dependency in your `Project.toml` — in which case, great, easy peasy, no problems there — or it has to be brought into scope by `include`ing its source file. If the module is in your own package, and not a dependency, then you have to `include` it somewhere. Correct me if I’m wrong but there is no third way to import a module.
I have no problem with `import`s — I think they’re a fine way of making an in-scope module and its items accessible. My issue is with how you get that module into scope in the first place. For that, you need `include`. And that’s where things get complicated.
---
### Author:  [@bert](https://discourse.julialang.org/u/bert)
#### Post date: [August 20, 2022, 3:25am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/35 "2022-08-20T03:25:44Z")
To make this point concrete, I’ll give a real-life example. Here’s a file in Makie: [Makie.jl/display.jl at master · JuliaPlots/Makie.jl · GitHub](https://github.com/JuliaPlots/Makie.jl/blob/master/src/display.jl) . Where do the following symbols come from?
- `@ffmpeg_env`
- `get_scene`
- `FileIO`
Answer: they’re made available by importing the relevant stuff into [Makie.jl/Makie.jl at master · JuliaPlots/Makie.jl · GitHub](https://github.com/JuliaPlots/Makie.jl/blob/master/src/Makie.jl) , `include`ing `display.jl` in that file, and then crossing their fingers that everything was wired up correctly across file boundaries.
Package authors shouldn’t have to keep track of the imports in one file in order to reason about a second file, _especially when the second file doesn’t tell you which file is the “first file”!_ If you’re editing `display.jl`, how are you supposed to find out the myriad places where `get_scene` is defined? Are you really expected to go through all of the `include`s in `Makie.jl` (the file) and check for the presence of that function definition?
I think this qualifies as [Action at a distance (computer programming) - Wikipedia](https://en.wikipedia.org/wiki/Action_at_a_distance_%28computer_programming%29) , which is universally recognized as a bad quality of a program.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [August 20, 2022, 3:41am UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/36 "2022-08-20T03:41:20Z")
> [@bert](#):
>
> Similarly, trying to chase down the origins of exports in the “main” file is a Herculean effort. You should never have to ctrl-f your entire workspace for the origin of a symbol that’s used in the current file – that’s a sign of bad design.
This is a consequence of multiple dispatch. If you see `export foo`, it means that you are exporting the variable `foo`. If `foo` is the name of a function, then you are exporting a _generic_ function. It’s not possible to export individual methods of functions, because they all have the same name. There could be a different `foo` method defined in every file in your package, so you have to “ctrl-f” to find all the different method definitions. That’s just the way it is—I don’t see a way around that. 🤷♂️
Generic functions are actually global objects, because they are defined by a global method table. So trying to define dependencies with Python-style `from pkg import foo` doesn’t really work.
Rather than resurrecting this old thread on a contentious topic, I recommend starting a new thread with concrete suggestions for improvement.
---
### Author:  [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [August 21, 2022, 6:58pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/37 "2022-08-21T18:58:56Z")
This pattern of separating code into different files that are then patched up with `include` seems to be common in the Julia package ecosystem, and I for one agree about it being harmful and ugly.
I think the situation would be better if `include`ing arbitrary code was disallowed, but `include`ing `module`s specifically was allowed.
But even that seems like only a partial solution.
---
### Author:  [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)
#### Post date: [August 22, 2022, 2:10pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/38 "2022-08-22T14:10:03Z")
I strongly disagree. IMO doing this is just bad design.
Adding methods to a function defined somewhere else is _mutating global state_, and this is a Bad Thing.
If I could, I would have this be prohibited by the compiler. And likewise prohibit type piracy, which is of similar character. (e.g. Rust prohibits the analogous “trait piracy”)
It is completely possible to use Python-style `from pkg import foo`, as the success of [FromFile](https://github.com/Roger-luo/FromFile.jl) has demonstrated. (Evidence: [1.1k new users in the past 30 days; for reference Pluto has had 5k new users in the past 30 days.](https://github.com/JuliaLang/julia/issues/4600#issuecomment-1221666767))
To those bitten by this issue, I would recommend using FromFile for your work. Or just using a tech stack other than Julia. (Personally I’ve now made the move to JAX, for this reason [and others](https://kidger.site/thoughts/jax-vs-julia/).)
---
### Author:  [@adienes](https://discourse.julialang.org/u/adienes)
#### Post date: [August 22, 2022, 3:59pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/39 "2022-08-22T15:59:30Z")
> [@patrick-kidger](#):
>
> Adding methods to a function defined somewhere else is _mutating global state_,
Would you consider it equally poor behavior to add such a method when one of the types is owned by the module in which that method is defined?
---
### Author:  [@patrick-kidger](https://discourse.julialang.org/u/patrick-kidger)
#### Post date: [August 22, 2022, 4:37pm UTC](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/40 "2022-08-22T16:37:59Z")
I definitely agree that having the module own the type is necessary. In most situations I think that additionally subtyping an abstract type would really be best practice.
That is, like the following example.
(Using FromFile rather than modules since that’s my preference, but mentally substitute the usual `include`/`import` boilerplate if you prefer.)
```julia
# foo.jl
abstract type AbstractFoo end
function accepts_foo(x::AbstractFoo)
error("Must define `accepts_foo` for type `$(typeof(x)) <: AbstractFoo`")
end
# foobar.jl
using FromFile: @from
@from "foo.jl" import AbstractFoo, accepts_foo
struct Foobar <: AbstractFoo
...
end
function accepts_foo(::Foobar)
...
end
```
This gives at least some discoverability to track what’s going on.
This analogous to overriding methods of a class in typical single-dispatch OO; just extending it out to multiple dispatch in the obvious way.
* * *
Don’t forget that proper importing of symbols helps with a lot more than just this. Even if you don’t know the method you at least know which _function_ is being called. Code exists in smaller namespaces so it’s harder to introduce various classes of bugs. Topologically sorting dependencies is no longer a burden placed upon the developer. And FromFile, at least, additionally handles deduplication (no `include`-ing a module twice). Etc. etc. you get the idea.
[Previous page](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835.md?page=1)
[Next page](https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835.md?page=3)