[ANN] PatModules.jl: a better module system for Julia

Package Announcements
57

I can’t be 100% sure because I am not looking at it from the same vantage point as you @patrick-kidger, but your thought pattern & solutions remind me alot of my own confusion when I started making modules/packages/namespaces in Julia

Header guards, as an example

“using header guards” is one of those things that got to me too - coming from a C/C++ background.

C/C++ vs Julia

But there are huge differences with include() in Julia: You don’t include header files in Julia! In C/C++, header files are used to tell the compiler how much space to allocate to objects, where data offsets are, and where to push data on the call stack before executing a function call.

That’s because C/C++ compiles machine code only, and doesn’t retain a table of that information anywhere (At least I don’t think it does when you turn off the debugger options).

Julia, on the other hand keeps this information around when it “compiles” its code (so I guess it does more than just compile). And this is a big difference. It’s also why Julia can do introspection so well (you can even access the original function code if you want).

When to use Julia include()s

First of all, you don’t include() external packages. You should only include() files that are in your current project or package (baring exceptional, possibly questionable circumstances). When you want to use external packages, you need to use import or using (but I won’t talk about those right now).

So right there, this is different from C/C++. Whereas in C/C++ you include header files that are built independently of your current project, in Julia, you only should be including code/files that you know are directly part of your project/package.

That means you can guarantee there is no double inclusion because you, as the developer, have control over this entire project/package codebase. Moreover, your code files need only be included once (it is not the same thing as C/C++ headers). That’s why you’ve noticed people often include() a bunch of subfiles from the same master file. Once include()-ed, that code is available to any other module loaded by Julia (There is no information hiding like there is in C/C++).

So how do Julia include()-s map to C/C++?

In reality, a Julia include() statement probably relates more closely to using the C/C++ linker:

g++ -o myexecutable first.o second.o third.o

In Julia, this would look like:

#myexecutable.jl

include("first.jl")
include("second.jl")
include("third.jl")
...

That’s because the Julia interpreter/compiler actually loads that code into memory and is ready to use it as soon as it processes the include() statement. It is not a preprocessor directive like it is in C/C++.

So how do you call code from another module, then?

Well, unlike C/C++, you don’t need to read in a header file to execute code, or build new structs in Julia. Once the code is loaded through an include() statement somewhere, your code can just execute it directly as long as it knows what the module path is:

#MyProject/src/subfile1.jl

#Why not define a submodule here? It doesn't have to the same name as the file.
#"module"s are really just namespaces, and are not tied to files in any way.
module SubA

module B #Again, why not another namespace here?
struct MyStruct
    x::Int; y::Int
end
end #module B

function dosomething(obj::B.MyStruct)
    #Do something
end

end #module SubA

#This function will be in the same namespace ("module") as whatever code
#called include("subfile1.jl"):
function dosomethingelse(obj::SubA.B.MyStruct)
    #Do something
end

#MyProject/src/MyProject.jl

#Julia projects & packages need to declare a module (namespace)
#with the same name as the project to function correctly.  Julia also
#expects you to create a file under src/ that has the same name
#as the project (thus /src/MyProject.jl).
module MyProject

include("subfile1.jl")

#Create a global object to store state:
glb_obj = SubA.B.MyStruct(3,5)

#Call functions that might have been written/loaded in other files:

SubA.dosomething(glb_obj)
dosomethingelse(glb_obj)

#...
end #module MyProject

Is that the same way we load external packges?

No, not exactly, code from external packages (even if not registered in Julia’s “General” registry) should be indirectly included with either the using or import statements:

#MyProject/src/MyProject.jl

#Julia projects & packages need to declare a module (namespace)
#with the same name as the project to function correctly.  Julia also
#expects you to create a file under src/ that has the same name
#as the project (thus /src/MyProject.jl).

module MyProject
using CSV

CSV.dosomething() #FYI: Doesn't actually exist.

#...
end #module MyProject

Note that Julia ensures that only a single instance of CSV is loaded - no matter how many modules (namespaces) call “using CSV”. It also ensures that ALL modules calling using CSV get a local pointer to the same CSV module (namespace), which gets loaded exactly once into memory (unless it needs to be re-evaluated for some reason - yeah. there are a bunch of exception cases, sorry.).

10 Likes