[ANN] PatModules.jl: a better module system for Julia

[MA_Laforge]

@patrick-kidger

I spent a fair bit of time searching around looking at existing solutions to this problem, and existing thoughts on how things may be improved:

Current recommended best practice 1
Current recommended best practice 2
Current way of performing relative imports
An example of what is done in existing major packages
A comparison to C++ (a language with the same basic issue)
#4600: a potential change, but not really a fix

With the general overview being that (a) the problem exists, (b) it has already been acknowledged, but © there are at present no good solutions.

I suggest looking at the following as well:
→ Dependencies of src files inside a package

I also creating a doc PR to address this issue (I don’t suggest reading the PR thread because it’s a bit hard to read).

Instead, I’ll insert the PR text directly here:

Inline text from PR

Julia ⇔ C/C++: Namespaces

• C/C++ namespaces correspond roughly to Julia modules.
• There are no private functions/variables/modules/… in Julia. Everthing is accessible
  through fully qualified paths (or relative paths, if desired).
• using MyNamespace::myfun (C++) corresponds roughly to import MyModule: myfun (Julia).
• using namespace MyNamespace (C++) corresponds roughly to using MyModule (Julia)
  • In Julia, only exported symbols are made available to the calling module.
  • In C++, only elements found in the included (public) header files are made available.
• Caveat: import/using keywords (Julia) also load modules (see below).
• Caveat: import/using (Julia) works only at the global scope level (modules)
  • In C++, using namespace X works within arbitrary scopes (ex: function scope).

Julia ⇔ C/C++: Module loading

• When you think of a C/C++ “library”, you are likely looking for a Julia “package”.
  • Caveat: C/C++ libraries often house multiple “software modules” whereas Julia
    “packages” typically house one.
  • Reminder: Julia modules are global scopes (not necessarily “software modules”).
• Instead of build/make scripts, Julia uses “Project Environments” (sometimes called
  either “Project” or “Environment”).
  • Build scripts are only needed for more complex applications
    (like those needing to compile, or download C/C++ executables ).
  • C/C++ code typically target more conventional applications, whereas Julia
    “Project Environments” provide a set of packages to experiment with particular problem
    spaces. Julia users typically use problem-specific “scripts” for this type of experimentation.
  • To develop a “conventional” application/project in Julia, you can initialize its root directory
    as a “Project Environment”, and house application-specific code/packages there.
    This provides good control over project dependencies, and future reproducibility.
  • Available packages are added to a “Project Environment” with the pkg> add tool
    (This does not load said package, however).
  • The list of available packages (direct dependencies) for a “Project Environment” are
    saved in its Project.toml file.
  • The full dependency information for a “Project Environment” is auto-generated & saved
    in its Manifest.toml file.
• Packages (“software modules”) available to the “Project Environment” are loaded with
  import or using.
  • In C/C++, you #include <moduleheader> to get object/function delarations, and link in
    libraries when you build the executable.
  • In Julia, whatever is loaded is available to all other loaded modules through its
    fully qualified path (no header file required).
  • Use import SomePkg: SubModule.SubSubmodule (Julia) to access package submodules.
• Directory-based package repositories (Julia) can be made available by adding repository
  paths to the Base.LOAD_PATH array.
  • Packages from directory-based repositories do not require the pkg> add tool prior to
    being loaded with import or using. They are simply available to the project.
  • Directory-based package repositories are the quickest solution to developping local
    libraries of “software modules”.

Julia ⇔ C/C++: Assembling modules

• In C/C++, .c/.cpp files are compiled & added to a library with build/make scripts.
  • In Julia, import [PkgName]/using [PkgName] statements load [PkgName].jl located
    in a package’s [PkgName]/src/ subdirectory.
  • In turn, [PkgName].jl typically loads associated source files with calls to
    include "[someotherfile].jl".
• include "./path/to/somefile.jl" (Julia) is very similar to
  #include "./path/to/somefile.jl" (C/C++).
  • However include "..." (Julia) is not used to include header files (not required).
  • Do not use include "..." (Julia) to load code from other “software modules”
    (use import/using instead).
  • include "path/to/some/module.jl" (Julia) would instantiate multiple versions of the
    same code in different modules (creating distinct types (etc.) with the same names).
  • include "somefile.jl" is typically used to assemble multiple files within the same
    Julia package (“software module”). It is therefore relatively straightforward to ensure
    file are included only once (No #ifdef confusion).

Julia ⇔ C/C++: Module interface

• C++ exposes interfaces using “public” .h/.hpp files whereas Julia modules export
  symbols that are intended for their users.
  • Often, Julia modules simply add functionality by generating new “methods” to existing
    functions (ex: Base.push!).
  • Developers of Julia packages therefore cannot rely on header files for interface
    documentation.
  • Interfaces for Julia packages are typically described using docstrings, README.md,
    static web pages, …
• Some developers choose not to export all symbols required to use their package/module.
  • Users might be expected to access these components by qualifying functions/structs/…
    with the package/module name (ex: MyModule.run_this_task(...)).

Julia ⇔ C/C++: Quick reference

Software Concept	Julia	C/C++
unnamed scope	begin … end	{ … }
function scope	function x() … end	int x() { … }
global scope	module MyMod … end	namespace MyNS { … }
software module	A Julia “package”	.h/.hpp files +compiled somelib.a
assembling software modules	SomePkg.jl: … import subfile1.jl import subfile2.jl …	$(AR) *.o ⇒ somelib.a
import software module	import SomePkg	#include <somelib> +link in somelib.a
module library	LOAD_PATH[], *Git repository, **custom package registry	more .h/.hpp files +bigger compiled somebiglib.a

* The Julia package manager supports registering multiple packages from a single Git repository.

* This allows users to house a library of related packages in a single repository.

** Julia registries are primarily designed to provide versionning & distribution of packages.

** Custom package registries can be used to create a type of module library.