How to organize modules in a composable way?

General Usage
1

Suppose I have a module module_init that defines a type init_struct, and I am now trying to make multiple modules in different projects that build on top of this init_struct. For example

./project_init/src/module_init.jl contains

module module_init
export init_struct,do_something
struct init_struct end
function do_something(init_struct)
     #do something
end
end

Then I have a few projects: eg project_sub1 and project_sub2 that have their code like.

./project_sub1/src/sub1_module.jl contains

module module_sub1
export sub1Struct,performSub1Action
include(raw"./project_init/src/module_init.jl")
using .module_init
struct sub1Struct{T}
     init::init_struct
     sub1::T
end
functon performSub1Action(input::sub1Struct{T}) where {T}
     do_something(input.init)
     # Do other actions for sub1
end
end

./project_sub2/src/sub2_module.jl contains

module module_sub2
export sub2Struct,performSub2Action
include(raw"./project_init/src/module_init.jl")
using .module_init
struct sub2Struct{T}
     init::init_struct
     sub2::T
end
functon performSub2Action(input::sub1Struct{T}) where {T}
     do_something(input.init)
     # Do other actions for sub1
end
end

This is not ideal because the composability is gone. I cannot do something like

include(“./project_init/src/module_init.jl”)
using .module_init
include("./project_sub1/src/sub1_module.jl")
using .module_sub1
init = init_struct()
sub1 = 3
data = sub1Struct(init,sub1)

Since the init_struct inside the sub1 module has a different namespace module_sub1.module_init

I can try to import and re-export, but then I will not be able to easily use module_sub1 and module_sub2 simultaneously.

How should I organize my code such that I am able to do something like

using .module_init
using .module_sub1
using .module_sub2

init = init_struct()
sub1 = 3
sub2 = 4
data1 = sub1Struct(init,sub1)
data2 = sub1Struct(init,sub2)

Similar to how I can use packages that is able to do this? Where I can do something like using module_init on both module module_sub1 and module_sub1.

Is packages the only way to do this? Also, what is the difference between a package and a module. I am unable to find the right keywords to search for documentation in this area.

2

(post deleted by author)

3

My first suggestion would be to refactor your code so you don’t need so many modules. My experience has been that submodules add significant complexity and cognitive overhead so I use them only for large projects where namespace collision is hard to avoid otherwise.

If refactoring is not possible then here is one way to accomplish what you want. Create three subprojects of your Main project, one for each module. Place the module code in those subproject directories. Then load the subproject packages with using rather than include.

Here’s how to create the subprojects starting from scratch. Enter the julia package manager by typing ] and then create the Main project directory:

@v1.12) pkg> generate Main
  Generating  project Main:
    Main\Project.toml
    Main\src\Main.jl

Activate the Main project:

(@v1.12) pkg> activate Main
  Activating project at `C:\Users\seatt\OneDrive\Documents\temp\Main`

Now create the three subprojects that will contain the three modules:

(Main) pkg> generate Main\module_init
  Generating  project module_init:
    Main\module_init\Project.toml
    Main\module_init\src\module_init.jl

(Main) pkg> generate Main\sub1Struct
  Generating  project sub1Struct:
    Main\sub1Struct\Project.toml
    Main\sub1Struct\src\sub1Struct.jl

(Main) pkg> generate Main\sub2Struct
  Generating  project sub2Struct:
    sub2Struct\Project.toml
    sub2Struct\src\sub2Struct.jl

Now link these subprojects as dependencies of the Main project:

(Main) pkg> dev Main\sub1Struct\\
   Resolving package versions...
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Project.toml`
  [95151458] + sub1Struct v0.1.0 `sub1Struct`
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Manifest.toml`
  [95151458] + sub1Struct v0.1.0 `sub1Struct`

(Main) pkg> dev Main\sub2struct\\
   Resolving package versions...
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Project.toml`
  [eadf8c1f] + sub2struct v0.1.0 `sub2struct`
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Manifest.toml`
  [eadf8c1f] + sub2struct v0.1.0 `sub2struct`

(Main) pkg> dev Main\module_init\\
   Resolving package versions...
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Project.toml`
  [c9778ed0] + module_init v0.1.0 `module_init`
    Updating `C:\Users\seatt\OneDrive\Documents\temp\Main\Manifest.toml`
  [c9778ed0] + module_init v0.1.0 `module_init`

Now check that everything is correct:

(Main) pkg> st
Project Main v0.1.0
Status `C:\Users\seatt\OneDrive\Documents\temp\Main\Project.toml`
  [c9778ed0] module_init v0.1.0 `module_init`
  [95151458] sub1Struct v0.1.0 `sub1Struct`
  [eadf8c1f] sub2struct v0.1.0 `sub2struct`

Now you can create julia code in the Main project directory like this:

using module_init
using module_sub1
using module_sub2

and your subpackages will be loaded correctly.

2 Likes
4

This is a good explanation of the difference between modules and packages https://stackoverflow.com/questions/75511154/whats-the-difference-between-modules-and-packages-in-julia

5

Thank you for the detailed explanation. If I consider the following case, say with a package like StaticArrays, I am able to do something like

using Pkg
Pkg.add("StaticArrays")

then I can do DependentModule1.jl

module DependentModule1
using StaticArrays
export StructType1
struct StructType1{T}
    field::SVector{3,T}
    enumeration::Int64
end
end

Similarly DependentModule2.jl

module DependentModule2
using StaticArrays
export StructType2
struct StructType2{T}
    field::SVector{4,T}
    enumeration::Int64
end
end

And now it is possible to do this in the repl

using StaticArrays
include("DependentModule1.jl")
include("DependentModule2.jl")
using .DependentModule1
using .DependentModule2

fieldStruct1 = SVector{3,Int64}((1,3,5))
fieldStruct2 = SVector{4,Int64}((0,2,4,6))
Struct1 = StructType1(fieldStruct1,0)
Struct2 = StructType2(fieldStruct2,1)

What is it about using Pkg that enables bypassing include(<<Path_to_StaticArrays.jl>>) and ability to remove the dot before using (using StaticArrays instead of using .StaticArrays)?

For the use case mentioned, module_init is expected to be something equivalent to StaticArrays but something that should stay internal/local to the computers we are using.

Is the typical working environment, just a big Project like the one in your comment? If so, when does the equivalent of include(“StaticArrays.jl”) happen. Does it happen the moment we install a package, or is it during the first invokation of using StaticArrays in the code either in a module or in the script. If using StaticArrays occurs multiple times in a code (in the module it calls and in the code), is the equivalent of include(“StaticArrays.jl”) done once or multiple times (mulitple times should cause errors since it keeps replacing the references right)? Does the Project use a completely different mechanism to load the code of the invoked packages.

6

You’re not supposed to include before each import in the first place, that’s why they’re separate actions. include evaluates a file’s text as Julia code into its parent module, while imports share names among existing modules. In your original example, module_sub1 and module_sub2 evaluated their own module_init expressions, in other words module_sub1.module_init and module_sub2.module_init are completely unrelated. Stripping the code down to just the module structure, you intended to do this instead:

# some master file

module module_init # your include call could do this
end

module module_sub1
  using ..module_init # dots indicate how many levels up to find the imported module
end

module module_sub2
  using ..module_init
end

println(module_sub1.module_init === module_sub2.module_init) # true

The package approach lets an environment track the modules, so no need for dotted names or evaluations to a master file. Personally I think it’s overkill for such little code, as is making those two modules separate to begin with. You’d know better how your code should be organized, though, this is clearly a minimal example.

7

Yeah, use packages instead of abusing include.

A module is just a namespace. That’s it.