# Organizing modules. Is it OK to organize a project using several modules in Julia?

**URL:** <https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535>\
**Category:** New to Julia\
**Created:** [February 12, 2020, 7:47pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535 "2020-02-12T19:47:29Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 7:47pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/1 "2020-02-12T19:47:29Z")

</div>

I’ve been trying to organize a small project using several modules. I am doing some experiments with a board game AI in this project. I have something like this:

BoardModule.jl:

```julia
module BoardModule
export Board, valid_moves
struct Board; ... end
valid_moves(board) = ...
end

```

GameRunner.jl:

```julia
module GameRunner
export play_game
include("BoardModule.jl")
using .BoardModule
play_game(player1, player2) = ...
end

```

PlayerModule.jl

```julia
module PlayerModule
export Player, move
include("BoardModule.jl")
using .BoardModule
struct Player; ... end
move(player::Player, board) = ...
end

```

The idea is straightforward, game rules are implemented by the Board module, a GameRunner can run games between two players, and players are structs (which may contain some private state) and an overloaded move function which is called to ask the player what move they want to make. There may be several types of Players each making moves using different algorithms: minmax, MCTS, a neural network, etc. These various players are the heart of what I want to experiment with.

The problem is that, as far as Julia is concerned, `GameRunner` and `PlayerModule` are not using the same `Board`, so they can’t communicate. `GameRunner` is using `GameRunner.BoardModule.Board`, and `PlayerModule` is using `PlayerModule.BoardModule.Board`; they’re different types, technically.

I don’t know how to reuse the `BoardModule`? (I’m afraid the answer will involve creating several different git repositories for a single project.)

* * *

Taking a step back now. When I search for “how to organize Julia modules” I find: [How to organise files and modules?](https://discourse.julialang.org/t/how-to-organise-files-and-modules/4842)

I would paraphrase the conclusion of that discussion to be: “ **It is ‘un-Julian’ to organize functions into several different modules.** ” Is this correct?

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 12, 2020, 7:54pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/2 "2020-02-12T19:54:03Z")

</div>

The Julia code itself is organised into different Modules: `Base`, `Core`, `Threads`, `LinearAlgebra` … So its definitely not “un-Julian” 🙂

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 8:05pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/3 "2020-02-12T20:05:53Z")

</div>

Good. 🙂

Any suggestion on how I could organize my small project then? I recognize it _might_ be more burdensome than ideal for the time being, since Julia is so young.

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [February 12, 2020, 8:23pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/4 "2020-02-12T20:23:30Z")

</div>

But Julia itself is huge. What do you hope to gain by making lots of modules here?

By all means write the code in different files, even put them in subfolders if it helps, but IHMO you should probably just `include` all of them in the main project’s module, each exactly once.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [February 12, 2020, 8:40pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/5 "2020-02-12T20:40:03Z")

</div>

I would suggest that you start playing around with module structures without using `include` or `export`, just to get a sense of how they work. Here’s an example of a module which is itself composed of 3 submodules, and in which the `GameRunner` and `PlayerModule` modules can both correctly access the `Board` type from `BoardModule`:

```julia
julia> module GameAI
         module BoardModule
           struct Board end
         end
         
         module GameRunner
           using ..BoardModule: Board

           run(b::Board) = println("running: $b")
         end
         
         module PlayerModule
           using ..BoardModule: Board
           
           move(b::Board) = println("moving on: $b")
         end
       end
Main.GameAI

julia> using .GameAI

julia> board = GameAI.BoardModule.Board()
Main.GameAI.BoardModule.Board()                                                                                                                                  
                                                                                                                                                                 
julia> GameAI.GameRunner.run(board)
running: Main.GameAI.BoardModule.Board()                                                                                                                         
                                                                                                                                                                 
julia> GameAI.PlayerModule.move(board)
moving on: Main.GameAI.BoardModule.Board()   

```

You can move any of those modules into their own files and replace the module definition with `include("module.jl")` as you see fit, but it’s not necessary in order to understand the module organization.

The other rule of thumb is: a `.jl` file should not be included more than once in your code.

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [February 12, 2020, 8:46pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/6 "2020-02-12T20:46:58Z")

</div>

@rdeits beat me to replying. I agree with what he says and there is overlap in the answers but there is some complementary info below:

It can be debated how many modules you want to break your project into. Without knowing more about the problem, I think the structure you outlined in the OP makes sense. Regardless of how you divide your code into modules I would definitely stop using `include` to import one module into another. I would take one of the following two approaches. The first is to have a master module for your project with the different functionality split off into submodules. I like this approach if the different submodules only really make sense as part of the game project and wouldn’t be useful on their own. This might have the following directory structure:

```julia
├── BoardModule
│ ├── Project.toml
│ └── src
│ └── BoardModule.jl
├── GameRunner
│ ├── Project.toml
│ └── src
│ └── GameRunner.jl
├── PlayerModule
│ ├── Project.toml
│ └── src
│ └── PlayerModule.jl
└── Project.toml

```

You can use `Pkg.generate` (see the [docs](https://julialang.github.io/Pkg.jl/v1/) for more info) to generate the Project.toml files for you. You can then run the command

```julia
> add /local-path-to/gameProject

```

from the Pkg repl mode to get access to gameProject and its submodules. Once you have your modules organized this way, you can refer to `BoardModule` from the other two submodules just as @rdeits mentioned, by writing

```julia
using ..BoardModule

```

You can also load `BoardModule` from outside the submodule just by doing `using gameProject.BoardModule` from the repl or in any other julia module.

The second approach is to just have the three modules be completely separate, each living in its own directory and having its own Project.toml file. Then you can add these modules to your julia environment individually by doing

```julia
> add /local-path-to-project/

```

from the Pkg repl mode.

edit: I originally had `> add gameProject` but of course that only works for registered packages. You need to run the add command with the local path to the project or the address of a remote git repository to add non-registered packages to your julia environment.

edit2: Above directory layout is wrong. Also you need to dev, not add if you don’t have a git repo in your local folder. See [post below](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/16) for updated directory structure if you want to use parent module and submodules.

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 8:56pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/7 "2020-02-12T20:56:32Z")

</div>

> [@improbable22](#):
>
> IMHO you should probably just `include` all of them in the main project’s module, each exactly once

Modules are synonymous with namespaces right? In other words, modules are the only mechanism we have to organize things into namespaces?

I would be hesitant to tell people “In Julia, most projects should be done entirely in a single namespace”. For example, if I drop that in the next Hacker News discussion about Julia, I believe it would harm rather than help the language.

Thus, while I accept that it is a valid opinion, it’s not an opinion I would want to push at “the Julia way”. Or rather, I wouldn’t discourage someone from using namespaces and instead encourage them to use a single namespace.

I would agree that Julia has some interesting features, and that putting everything in a single namespace has unique advantages in Julia (and some disadvantages, pros and cons, etc).

* * *

I ended up solving my problem by modifying the [`LOAD_PATH`](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1), which allows me to do `using BoardModule` without having to `include` it. (See: [Code Loading · The Julia Language](https://docs.julialang.org/en/v1/manual/code-loading/#Package-directories-1)) I also don’t need any additional `Package.toml` files. I’m not saying this is the _best_ way, but it seems like a good way.

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 9:05pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/8 "2020-02-12T21:05:49Z")

</div>

> [@Pbellive](#):
>
> > add /local-path-to/gameProject

Do you mean `Pkg.add("/local-path-to/gameProject")`? I know about the convenient way to access it with `]` in the REPL, I just want to be clear if you are referring to the `Pkg.add` command on this line?

I thought anything I ran `Pkg.add` on needed to be in it’s own git repository? Maybe I’m wrong?

In your example, would you have each of the 3 “Projects” in their own git repo? Would you have 1 git repo (holding everything in your tree diagram), or 3 git repos?

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [February 12, 2020, 9:09pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/9 "2020-02-12T21:09:27Z")

</div>

Sure, it’s a judgement call, and I only offered an opinion based on what I could see in your question. Many of us are guilty of premature optimisation for speed, and this looked like the equivalent sin for organisational complexity! But we learn things along the way all the same…

My example on HN was [Optim.jl](https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/Optim.jl) which, despite being quite a large project, hasn’t bothered with sub-modules at all.

I believe you can `] dev path/to/GameProject/` without it being a git repository. So long as it has `/src/GameProject.jl` and `/Project.toml` inside. Which `] generate GameProject` will create.

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [February 12, 2020, 9:18pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/10 "2020-02-12T21:18:47Z")

</div>

You can use the same syntax as in the ] Repl also in scripts:

```julia
using Pkg
pkg"add /path/MyProject"

```

is equivalent to

```julia
using Pkg
Pkg.add("/path/MyProject")

```

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 9:20pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/11 "2020-02-12T21:20:19Z")

</div>

> [@lungben](#):
>
> You can use the same syntax as in the ] Repl also in scripts:
> 
> ```julia
> using Pkg
> pkg"add /path/MyProject"
> 
> ```

I didn’t know that. Good tip. 👍

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [February 12, 2020, 9:30pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/12 "2020-02-12T21:30:42Z")

</div>

> [@Pbellive](#):
>
> > > add /local-path-to/gameProject
> 
> Do you mean `Pkg.add("/local-path-to/gameProject")` ? I know about the convenient way to access it with `]` in the REPL, I just want to be clear if you are referring to the `Pkg.add` command on this line?

Yes, running `> add x` from the Pkg repl mode is the same as running `Pkg.add("x")` from the regular repl or in a script.

> [@DevJac](#):
>
> I thought anything I ran `Pkg.add` on needed to be in it’s own git repository? Maybe I’m wrong?

No, you don’t need to have any git repositories at all to use `Pkg.add`, you just need a Project.toml file. You also don’t need a single git repo per module. My example above works with a single git repository for the project or without any git repo at all.

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 12, 2020, 9:35pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/13 "2020-02-12T21:35:33Z")

</div>

You can use modules, it is not “un-Julian” but you don’t need to, unless you find it necessary to encapsulate variables and functions from each other and from outer access.

In Julia you cannot completely hide scopes from outer access but you can impede external access and avoid namespace collisions by using modules.

Decades ago I used Modula-2, and it was very strict with exporting and namespaces and so on, but now I like less constraints on namespace organisation.

Therefore I would start with a flat project organisation (one module) and add further modules only if necessary. For a small or medium-size manageable project I would suggest you think first how to organise your functionality into \*.jl files, which you `include` into your main `GameXY.jl` file, defining the main module.

This will change sometimes during development. Only then – if necessary or convenient – I would introduce submodules.

Julia has a terrific feature: multiple dispatch, which allows you to reuse function names for similar purposes and with different arguments. And therefore you should try to exploit more multiple dispatch and use less modules.

Therefore from experience and contrary to early software-engineering philosophy (Modula-2 …) I would avoid prematurely modularising my project.

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 9:48pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/14 "2020-02-12T21:48:53Z")

</div>

```shell
➜ gobot git:(master) ✗ tree .
.
├── Manifest.toml
├── packages
│ └── BoardM
│ ├── Project.toml
│ └── src
│ └── BoardM.jl
├── Project.toml
└── src
    ├── GameRunner.jl
    ├── mcbot.jl
    └── nnbot.jl

4 directories, 7 files
➜ gobot git:(master) ✗ julia --project --banner=no
julia> using Pkg

julia> Pkg.add("/home/buttons/Code/julia/gobot/packages/BoardM")
ERROR: /home/buttons/Code/julia/gobot/packages/BoardM is not a valid packagename.
The argument appears to be a URL or path, perhaps you meant `Pkg.add(PackageSpec(url="..."))` or `Pkg.add(PackageSpec(path="..."))`.
Stacktrace:
    ...

julia> Pkg.add(PackageSpec(path="/home/buttons/Code/julia/gobot/packages/BoardM"))
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `/home/buttons/Code/julia/gobot/packages/BoardM`
ERROR: Git repository not found at '/home/buttons/Code/julia/gobot/packages/BoardM'
    ...

```

It doesn’t seem to work for me?

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [February 12, 2020, 10:05pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/15 "2020-02-12T22:05:52Z")

</div>

I still think you need to dev not add:

```julia
(v1.3) pkg> generate ~/trash/One
Generating project One:
    ~/trash/One/Project.toml
    ~/trash/One/src/One.jl

(v1.3) pkg> add ~/trash/One
   Cloning git-repo `/Users/me/trash/One`
ERROR: Git repository not found at '/Users/me/trash/One'

(v1.3) pkg> dev ~/trash/One
 Resolving package versions...
  Updating `~/trash/sklfjhs/Project.toml`
  [971f93ee] + One v0.1.0 [`~/trash/One`]
  Updating `~/trash/sklfjhs/Manifest.toml`
  [971f93ee] + One v0.1.0 [`~/trash/One`]

julia> using One
[Info: Precompiling One [696393eb-e9a0-4ed0-9ad8-d55bac9cf5e1]

julia> One.greet()
Hello World!

```

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [February 12, 2020, 10:16pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/16 "2020-02-12T22:16:19Z")

</div>

Wow, my post was pretty sloppy. Sorry about that. @improbable22 is right, you have to dev, not add if you are just using a local path without a git repo. dev from the pkg repl mode is of course equivalent to `Pkg.develop`

Additionally, the directory structure I set up was just flat wrong. I guess that’s what happens when I rush and don’t actually test what I’m suggesting. The following directory structure works:

```julia
├── Project.toml
└── src
    ├── BoardModule
    │ └── BoardModule.jl
    ├── gameProject.jl
    ├── GameRunner
    │ └── GameRunner.jl
    └── PlayerModule
        └── PlayerModule.jl

```

Note that if you’re doing one parent module with submodules only the parent module needs a Project.toml.

Then `gameProject.jl` can contain

```julia
module gameProject

include("BoardModule/BoardModule.jl")
include("PlayerModule/PlayerModule.jl")
include("GameRunner/GameRunner.jl")

end # module

plus any other code you want in the top level. Then, say BoardModule. jl is

```

module BoardModule

export board\_greeter

board\_greeter() = “Hello World! I’m a game board”

end # module

```julia

Then GameRunner.jl could be 

```

module GameRunner

export game\_greeter

import …BoardModule: board\_greeter

game\_greeter() = println(“Hi, I’m a game. My board also says hello: $(board\_greeter())”)

end

```julia

This setup I actually implemented on my local machine and tested that it works.
```

---

<div class="post-metadata">

**Author:** ![DevJac](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@DevJac](https://discourse.julialang.org/u/DevJac)\
**Post date:** [February 12, 2020, 10:17pm UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/17 "2020-02-12T22:17:25Z")

</div>

You’re right. It works if I use `dev`. Thank you.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [February 13, 2020, 2:27am UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/18 "2020-02-13T02:27:51Z")

</div>

When you use the “main module with sub-modules” approach (one `Project.toml`) the directory / file structure does not matter any more.

You can have any number of files per module (or modules per file) in any location.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 13, 2020, 7:34am UTC](https://discourse.julialang.org/t/organizing-modules-is-it-ok-to-organize-a-project-using-several-modules-in-julia/34535/19 "2020-02-13T07:34:08Z")

</div>

> [@DevJac](#):
>
> I would be hesitant to tell people “In Julia, most projects should be done entirely in a single namespace”. For example, if I drop that in the next Hacker News discussion about Julia, I believe it would harm rather than help the language.
> 
> Thus, while I accept that it is a valid opinion, it’s not an opinion I would want to push at “the Julia way”. Or rather, I wouldn’t discourage someone from using namespaces and instead encourage them to use a single namespace.

I think it is best to be practical about this: when code matures and becomes very complex, it may be worthwhile to use namespaces with a finer granularity (but this is frequently close to the point where making a separate package is the best way to go).

But when you are just starting a new project, going overboard with namespaces (and even file layouts) is usually just a distraction. Keep in mind that Julia is a very expressive language, which results in quite compact code, and it has other features that provide some level of orthogonality. Eg if a set of methods operates on some particular type, you could consider moving them to a submodule, but you don’t need to because there is little possibility for confusion.

Code moves fast and may be refactored many times before it settles. Its best to keep things simple.