I’ve been trying to organize a small project using several modules. I am doing some experiments with a board game AI in this project. I have something like this:
BoardModule.jl:
module BoardModule
export Board, valid_moves
struct Board; ... end
valid_moves(board) = ...
end
GameRunner.jl:
module GameRunner
export play_game
include("BoardModule.jl")
using .BoardModule
play_game(player1, player2) = ...
end
PlayerModule.jl
module PlayerModule
export Player, move
include("BoardModule.jl")
using .BoardModule
struct Player; ... end
move(player::Player, board) = ...
end
The idea is straightforward, game rules are implemented by the Board module, a GameRunner can run games between two players, and players are structs (which may contain some private state) and an overloaded move function which is called to ask the player what move they want to make. There may be several types of Players each making moves using different algorithms: minmax, MCTS, a neural network, etc. These various players are the heart of what I want to experiment with.
The problem is that, as far as Julia is concerned, GameRunner and PlayerModule are not using the same Board, so they can’t communicate. GameRunner is using GameRunner.BoardModule.Board, and PlayerModule is using PlayerModule.BoardModule.Board; they’re different types, technically.
I don’t know how to reuse the BoardModule? (I’m afraid the answer will involve creating several different git repositories for a single project.)
Any suggestion on how I could organize my small project then? I recognize it might be more burdensome than ideal for the time being, since Julia is so young.
But Julia itself is huge. What do you hope to gain by making lots of modules here?
By all means write the code in different files, even put them in subfolders if it helps, but IHMO you should probably just include all of them in the main project’s module, each exactly once.
I would suggest that you start playing around with module structures without using include or export, just to get a sense of how they work. Here’s an example of a module which is itself composed of 3 submodules, and in which the GameRunner and PlayerModule modules can both correctly access the Board type from BoardModule:
julia> module GameAI
module BoardModule
struct Board end
end
module GameRunner
using ..BoardModule: Board
run(b::Board) = println("running: $b")
end
module PlayerModule
using ..BoardModule: Board
move(b::Board) = println("moving on: $b")
end
end
Main.GameAI
julia> using .GameAI
julia> board = GameAI.BoardModule.Board()
Main.GameAI.BoardModule.Board()
julia> GameAI.GameRunner.run(board)
running: Main.GameAI.BoardModule.Board()
julia> GameAI.PlayerModule.move(board)
moving on: Main.GameAI.BoardModule.Board()
You can move any of those modules into their own files and replace the module definition with include("module.jl") as you see fit, but it’s not necessary in order to understand the module organization.
The other rule of thumb is: a .jl file should not be included more than once in your code.
@rdeits beat me to replying. I agree with what he says and there is overlap in the answers but there is some complementary info below:
It can be debated how many modules you want to break your project into. Without knowing more about the problem, I think the structure you outlined in the OP makes sense. Regardless of how you divide your code into modules I would definitely stop using include to import one module into another. I would take one of the following two approaches. The first is to have a master module for your project with the different functionality split off into submodules. I like this approach if the different submodules only really make sense as part of the game project and wouldn’t be useful on their own. This might have the following directory structure:
You can use Pkg.generate (see the docs for more info) to generate the Project.toml files for you. You can then run the command
> add /local-path-to/gameProject
from the Pkg repl mode to get access to gameProject and its submodules. Once you have your modules organized this way, you can refer to BoardModule from the other two submodules just as @rdeits mentioned, by writing
using ..BoardModule
You can also load BoardModule from outside the submodule just by doing using gameProject.BoardModule from the repl or in any other julia module.
The second approach is to just have the three modules be completely separate, each living in its own directory and having its own Project.toml file. Then you can add these modules to your julia environment individually by doing
> add /local-path-to-project/
from the Pkg repl mode.
edit: I originally had > add gameProject but of course that only works for registered packages. You need to run the add command with the local path to the project or the address of a remote git repository to add non-registered packages to your julia environment.
edit2: Above directory layout is wrong. Also you need to dev, not add if you don’t have a git repo in your local folder. See post below for updated directory structure if you want to use parent module and submodules.
Modules are synonymous with namespaces right? In other words, modules are the only mechanism we have to organize things into namespaces?
I would be hesitant to tell people “In Julia, most projects should be done entirely in a single namespace”. For example, if I drop that in the next Hacker News discussion about Julia, I believe it would harm rather than help the language.
Thus, while I accept that it is a valid opinion, it’s not an opinion I would want to push at “the Julia way”. Or rather, I wouldn’t discourage someone from using namespaces and instead encourage them to use a single namespace.
I would agree that Julia has some interesting features, and that putting everything in a single namespace has unique advantages in Julia (and some disadvantages, pros and cons, etc).
I ended up solving my problem by modifying the LOAD_PATH, which allows me to do using BoardModule without having to include it. (See: Code Loading · The Julia Language) I also don’t need any additional Package.toml files. I’m not saying this is the best way, but it seems like a good way.
Do you mean Pkg.add("/local-path-to/gameProject")? I know about the convenient way to access it with ] in the REPL, I just want to be clear if you are referring to the Pkg.add command on this line?
I thought anything I ran Pkg.add on needed to be in it’s own git repository? Maybe I’m wrong?
In your example, would you have each of the 3 “Projects” in their own git repo? Would you have 1 git repo (holding everything in your tree diagram), or 3 git repos?
Sure, it’s a judgement call, and I only offered an opinion based on what I could see in your question. Many of us are guilty of premature optimisation for speed, and this looked like the equivalent sin for organisational complexity! But we learn things along the way all the same…
My example on HN was Optim.jl which, despite being quite a large project, hasn’t bothered with sub-modules at all.
I believe you can ] dev path/to/GameProject/ without it being a git repository. So long as it has /src/GameProject.jl and /Project.toml inside. Which ] generate GameProject will create.
Yes, running > add x from the Pkg repl mode is the same as running Pkg.add("x") from the regular repl or in a script.
No, you don’t need to have any git repositories at all to use Pkg.add, you just need a Project.toml file. You also don’t need a single git repo per module. My example above works with a single git repository for the project or without any git repo at all.
You can use modules, it is not “un-Julian” but you don’t need to, unless you find it necessary to encapsulate variables and functions from each other and from outer access.
In Julia you cannot completely hide scopes from outer access but you can impede external access and avoid namespace collisions by using modules.
Decades ago I used Modula-2, and it was very strict with exporting and namespaces and so on, but now I like less constraints on namespace organisation.
Therefore I would start with a flat project organisation (one module) and add further modules only if necessary. For a small or medium-size manageable project I would suggest you think first how to organise your functionality into *.jl files, which you include into your main GameXY.jl file, defining the main module.
This will change sometimes during development. Only then – if necessary or convenient – I would introduce submodules.
Julia has a terrific feature: multiple dispatch, which allows you to reuse function names for similar purposes and with different arguments. And therefore you should try to exploit more multiple dispatch and use less modules.
Therefore from experience and contrary to early software-engineering philosophy (Modula-2 …) I would avoid prematurely modularising my project.
Wow, my post was pretty sloppy. Sorry about that. @improbable22 is right, you have to dev, not add if you are just using a local path without a git repo. dev from the pkg repl mode is of course equivalent to Pkg.develop
Additionally, the directory structure I set up was just flat wrong. I guess that’s what happens when I rush and don’t actually test what I’m suggesting. The following directory structure works:
Note that if you’re doing one parent module with submodules only the parent module needs a Project.toml.
Then gameProject.jl can contain
module gameProject
include("BoardModule/BoardModule.jl")
include("PlayerModule/PlayerModule.jl")
include("GameRunner/GameRunner.jl")
end # module
plus any other code you want in the top level. Then, say BoardModule. jl is
module BoardModule
export board_greeter
board_greeter() = “Hello World! I’m a game board”
end # module
Then GameRunner.jl could be
module GameRunner
export game_greeter
import …BoardModule: board_greeter
game_greeter() = println(“Hi, I’m a game. My board also says hello: $(board_greeter())”)
end
This setup I actually implemented on my local machine and tested that it works.
I think it is best to be practical about this: when code matures and becomes very complex, it may be worthwhile to use namespaces with a finer granularity (but this is frequently close to the point where making a separate package is the best way to go).
But when you are just starting a new project, going overboard with namespaces (and even file layouts) is usually just a distraction. Keep in mind that Julia is a very expressive language, which results in quite compact code, and it has other features that provide some level of orthogonality. Eg if a set of methods operates on some particular type, you could consider moving them to a submodule, but you don’t need to because there is little possibility for confusion.
Code moves fast and may be refactored many times before it settles. Its best to keep things simple.
