Best practices for structuring larger projects

General Usage
1

I have a project that is becoming unwieldy in size and I’d like to break it up into modules. It doesn’t yet have parts that could be factored out as packages but I don’t want to rule out that it is heading that way. It has, however, reached a point at which I could have some submodules depend on others. However, there doesn’t seem to be a proper way to organize such projects in Julia.

The pattern I see (e.g., in Debug.jl) is to have a single “central” module-file that includes files that (sometimes) contain submodules. In that approach, dependencies between the submodules seem to be encoded in the order in which they are included in the central file. This strikes me as neither very robust/resilient nor scalable. Is there any best practice guide or an example project that people would refer to to illustrate how to structure projects like mine?

1 Like
2

A lot of package ecosystems use a “Base package” for its type hierarchy and for defining common functions, and then use satellite packages which build the true functionality off of that. I describe it a bit in more detail here:

You should look at things like JuliaDiff, JuliaDiffEq, etc. for more details.

1 Like
3

Well, this is interesting but not quite solving my problem: the complexity of my package doesn’t stem from the need to provide multiple (optimized) implementations for the same interface. It stems from the fact that I have many different components that have to work together. I could break them up into multiple packages but the “satellite” packages would be pretty much useless which seems to defy the point of packages.

2 Likes
4

Maybe this could help:

https://github.com/simonster/Reexport.jl

if you want to have submodules and not worry much about where symbols are from.

5

Also interesting but also not solving my problem :-). This isn’t really making it any easier to structure my project in multiple files. I guess we could use the inspiration and build a macro-package to provide a higher-level import mechanism. Much the way Javascript folks ended up with RequireJS we might end up with a hacky quasi-standard. However, the JS folks have the excuse that their language is 20 years old. I think Julia needs a good project structure/submodule story soon.

1 Like
6

Maybe the structure of https://github.com/JuliaDocs/Documenter.jl is useful to look at?

7

You might take a look at the current version of LightGraphs - we broke out related functions into separate files. We’re moving away from this (towards submodules) in the next major update, but the existing structure has served us well until now.

8

LightGraphs is a nice example (potentially the cleanest project I’ve seen while looking into this). I still don’t like the idea of having a single file in which all the “subfiles” are included though.

Do you have any efforts on the way towards submodules that I could look at?

9

Check out these github issues. https://github.com/JuliaLang/julia/issues?q=is%3Aopen+is%3Aissue+label%3Amodules

Any of them seem like they would help?

10

I still don’t like the idea of having a single file in which all the “subfiles” are included though.

Would you like to expand on that? What is is that you object to?
It seems to work well for many Julia packages.

11

LightGraphsExtras has a modular structure that might be of interest.

1 Like