What if the module dependency graph is not a tree? Can one have a network of modules calling each other? What about e.g. some utility functions that are used by different modules at different levels? Should they be include’d in each module? (And be compiled several times in different namespaces then?)
julia> module A
using Plots
end
Main.A
julia> module B # already faster
using Plots
end
Main.B
julia> @time p = A.plot(rand(10))
2.532125 seconds (3.23 M allocations: 194.841 MiB, 8.56% gc time, 59.41% compilation time)
julia> @time p = B.plot(rand(10))
0.001114 seconds (1.09 k allocations: 72.070 KiB)
Oh, sorry, probably I didn’t get the point. No, you should not include files more than once. If a utility is used in several modules, the best is to either include it once, at the top level, or even better, make it a package and use the package wherever needed.
If you are really thinking about code reuse, use and abuse of packages. That is by far the best way, for you and for others, to use the code. Packages can be minimal and lightweight, like this: https://github.com/SciML/IfElse.jl/blob/master/src/IfElse.jl
In general, I understand that the intended way of code reuse is creating packages, not modules. But, like the author of the question referenced in my posting, I’m concerned about creating lots of packages, debugging them separately, putting them to separate repos, etc. My experience with Node.js where a similar approach is used makes me skeptical about that. But I agree that using packages would probably allow a network of packages if the dependency graph is acyclic.
My approach is: include files once at the top level of the module and, when some of these files become interesting enough that I think I (or others) can use those functions in a separate project, split that into a package.
In my case, that is a very nice approach, because sometimes the “user base” is just my graduate students, and even for that splitting the code into a package is very, very convenient. (not necessarily I register every one of these packages, but if they are going to be dependencies of other projects, I do).
I do not use submodules in general. When I feel the need for that is when I split the packages. Thus, I have something like:
module MyPackage
### include("./somefunctions.jl")
somefunction1() = 1
somefunction2() = 2
###
include("otherfunctions.jl") # these may depend on "somefunctions"
mypackagefunction() = somefunction1() + otherfunction1()
end
Now, somefunctions.jl turned out to be interesting to be used in other packages, etc. Then I turn it into a package which exports the original somefunctions functions, and MyPackage is updated to be:
module MyPackage
using SomeFunctions
include("otherfunctions.jl") # these may depend on "somefunctions"
mypackagefunction() = somefunction1() + otherfunction1()
end
In other words, packages remain essentially flat, always.
Of course I do not claim that this is the best and only solution in every case, but it is a common and convenient pattern in Julia packages, as far as I can see.
(all strategies have advantages and drawbacks, following these stuff is nice because one learns many ways to think about the same problems, even if we are not using the ideas directly).
Just scanned it briefly… IMO it’s really too strong to say “modules are not giving you any value”. Maybe
‘modules’ is somewhat a misleading name. It probably makes some people (me too) think that modules are not namespaces, but units of compilation, as in Java.
I don’t have strong opinions on that because my background was in Fortran, and somewhat poorly written Fortran, so my use of any kind of “modules” was not very sophisticated, my transition to Julia was simple in that sense.
I do think that the “modules” in Python have some strengths in the sense of being self-contained and reusable. Julia modules do not play the same role as those, so the way to use modules is not the same as people are used to in Python. (I have nothing to say about modules in any other language).
My experience is that the “flat package” strategy is good and simple enough (you have to sort the “include” order by yourself, and that is the main critic as I see). If dependencies and reuse get more complicated, an intermediate step would be those sort of “self contained” modules, but really that is a patch in the midterm of having a package, which is the actual code unit that can really be used and reused with all possible flexibility.
Modules are units of compilation (well, technically a single function is the smallest unit of compilation) as well as namespaces. It’s just that in most cases, having a lot of submodules doesn’t give you a lot of benefit (as of today, there’s no data hiding like with private in java).
If the module is useful outside of the current project, making it a package of its own makes sense because then you don’t have to carry around a potentially larger than necessary dependency (think like project C depends on the submodule B of package A - if B is not a package on its own, you’ll have to carry around A as well).
If the module is only useful in the context of the current project, more often than not, building deep dependency chains of modules leads to a lot of A.B.C.D, which vastly hinders readability/reusability.
Note that this doesn’t mean that you shouldn’t use any submodules - if it makes sense to encapsule some functionality in a module, do it! An example I’m thinking of where it could make sense would be a game engine, with a module structure like this:
Engine
Audio
Input
Graphics
Network
In those cases, you don’t have a lot of common functionality, it’s not going to be used outside of the engine and parts only interact sparsly, so it can make sense to split it this way. Having a module per struct (which would be the java equivalent) is a bit too verbose though.
Thank you for that feedback! May I ask, how did you come across this topic? Modules are introduced fairly late, well after functions, methods, types, structs and constructors, so I’m wondering how this and how it’s done in julia could be introduced better. The section on submodules is only mentioned in passing, not going into a lot of detail compared to the other sections. Is there anything we can change in the documentation to make this easier?
In my really humble opinion, for the first presentation of the modules, it’s more important to present them as namespaces.
Yes, I understand that modules are precompliled, but AFAIK this compilation does not produce any deliverables that could be used separately. (E.g. like jar files.) So I see modules as namespaces first of all. I also use old good TCL language in my practice. Its concept of namespaces is somewhat similar to Julia. (But different nevertheless.)
In Java, namespaces and modules are orthogonal concepts. In Julia, they are nearly the same. This looks like an easy way to misunderstanding.
Hm… The manual section introducing modules already starts out with namespace management That modules are precompiled and cached really is more of an implementation detail (though that may change in the future, if/when static compilation is a thing, which it seems to be somewhere on the horizon). That’s why I corrected myself above to “functions are the smallest unit of compilation”, even though that’s not quite correct either - it requires the existence of a struct or other stuff from the same namespace to do anything useful.
Maybe there’s some wording we could change to make this distinction more explicit?
Well, I’m not sure if the problem is in the wording… IMO the main problem is a mix of high-level (namespaces) and low-level (compilation) concepts. Maybe low-level concepts to be put to another chapter.
That modules are precompiled and cached really is more of an implementation detail (though that may change in the future, if/when static compilation is a thing, which it seems to be somewhere on the horizon). That’s why I corrected myself above to “functions are the smallest unit of compilation”, even though that’s not quite correct either - it requires the existence of a struct or other stuff from the same namespace to do anything useful.