I see what you mean. The approach you took seems to me a sound one to take despite the drawbacks which I believe are present everywhere. Furthermore, this reminds me of - if I remember correctly - The Cathedral and the Bazaar by Eric Raymond where he posits that a way for devs to stay engaged and happy is to be owners of their software (by owners I mean sole developers with some autonomy and ideally, copyright). In that case it was more like moving the module inter communication from internal API (case for big corporations and software systems) to external/public APIs relying on some OS provided protocol i.e. file IO for UNIX for example. I wonder if this may be a barrier for development of very large codebases w internal APIs, as @MacKa 's observations seem to point to.
As an aside, this piqued my interest and I just finished reading it. What a great essay! The Julia project is clearly set up as a Bazaar, and I think that comes with some surprises to certain users. Areas without polish are really just problems waiting to be solved by whomever is able
Sorry, late for the discussion, but let me give my two cents:
The best option is a mix of options 2 and 4, which by themselves are not really options I see someone considering.
A submodule should only be included where it appears in the hierarchy for an external user. If a package Graphs.jl has Graphs.Algorithms then Algorithms.jl should be included inside Graphs.jl, if it is Graphs.Utilities.Algorithms then then Algorithms.jl should be included inside Utilities.jl which is included inside Graphs.jl. In other words, include should be only used to build the unique hierarchy of the modules in the project, it is not a tool to load code that will just be called. What however, if Graphs.Experimental.Search needs to use some methods from Graphs.Utilities.Algorithms? Simple, in Search.jl you do:
import ...Utilities.Algorithms
See Modules · The Julia Language about the relative import syntax.
I structured a rather large package with all my code from my PhD following this system. Here is the main file: https://github.com/henriquebecker91/GuillotineModels.jl/blob/master/src/GuillotineModels.jl and here is the file of GuillotineModels.PPG2KP: https://github.com/henriquebecker91/GuillotineModels.jl/blob/master/src/PPG2KP/PPG2KP.jl Note that I did also started a new subfolder for each submodule that had submodules themselves to avoid clashes like two Algorithm.jl in a Graphs.Utilities.Algorithms and a Graphs.Experimental.Algorithms.
One common pain point I find with nested modules is that each time you need to redefine all using and import dependencies. For my use cases it is pretty obvious that a nested module should have at least the same dependencies and namespace visibility as its parent. I see how this could create much noise in multi nested cases, but in my opinion if you don’t want this, you probably don’t even need nested modules. (Happy to hear counter use-cases)
Such a feature could probably be implemented with a macro, but unfortunately macros don’t play well with LSP, since this aspect was already mentioned above.
You could just refer to the parent module’s namespace instead of redoing the import.
julia> module A
export helloworld
helloworld() = println("Hello World")
end
Main.A
julia> module B
using ..A
module C
import ..B
foo() = B.helloworld()
end
end
Main.B
julia> B.C.foo()
Hello World
If you want exactly the same imports as the parent module, I’m less clear on why one would need a nested module to begin with as well.
Also for some Reexport.jl may be useful.
You are right, @reexport can already do this.
A small example for reference, since I already tested it:
module A
using Reexport
@reexport using Graphs
export s3
s3() = SimpleGraph(3)
s5() = SimpleGraph(5)
module B
# using Graphs # --> this line is not needed anymore
using ..A
s1s3s5() = [SimpleGraph(1), s3(), A.s5()]
end
end
On thinking this twice, @reexport will nicely propagate the dependencies in inner modules, but it will also pollute the user’s session in case the user makes a using A. So I wouldn’t prefer it for more complex (and realistic) scenarios. I think the unique solution is to write the boilerplate code (e.g. the using Graphs etc.).
The question is why nested modules start with a 0 dependencies (e.g. Graphs) and don’t inherit their parent’s ? Or, to be more dramatic, what features do nested modules actually bring into the table, that are developer attractive ? (other than logically categorizing structs/functions which is mostly user-focused)
I think such a behavior may create dependency problems (i.e. create depedency cycles) and would be most unwelcome. The rule is namespace isolation for safety. Therefore the developer is required to be explicit or intentional.
For example, in clean architecture patterns (i.e. where functional dependencies run only towards the higher level modules), one should only import the interfaces (in Julia, import functions to be overloaded for example) that the parent module exposes as such (although such thing is not enforced in Julia; happens at package level with all the modules ending in <SomethingSomething>Base). This makes it easier to work independently on larger codebases as only changes to the interface will break lower level functionality and changes to lower level code will not affect higher level functionality. More on this stuff here. I am not sure if this is the case - but the Pkg dependency resolver should handle this sort of things at higher level and I remember some efforts years ago towards this - probably long solved by now.
Create an inner module that @reexports what you want. Use that module for all your internal modules.
The main feature for me is that it helps to segment my code into parts that have distinct group of dependencies. The other feature is that it creates a namespace.
Overall, it is better that Julia imports less by default. It is then much easier to customize Julia to do exactly what you want.