Modular local projects and `include(...)`

New to Julia
, ,
1

Are local files/modules ever immediately importable with using , or is include always required?

For example, with a structure like this:

Project.toml
src/                # Source for a local library `MyLib`
    MyLib.jl        # contains `module MyLib`
    SomeModule.jl   # contains `module SomeModule`
    SomeModule/
        Foo.jl
        Bar.jl
runnable_file_1.jl  # these two rely on MyLib.jl
runnable_file_2.jl

Is all of the following necessary?

  • runnable_file_1 and 2 have include("src/MyLib.jl"); using MyLib
  • MyLib.jl has include("SomeModule.jl"); export SomeModule
  • SomeModule.jl has include("SomeModule/Foo.jl"); export Foo and the same for bar

It seems to me that there might be a way for directory structure to create module structure without manually including then exporting everything - such that a top level runnable file could simply using MyLib.SomeModule.Foo without needing the include statement at all (ala Python with __init__.py or Rust with pub mod).

Is something simpler possible? Alternatively, is there a better structure for a local package than what I describe?

2

This question comes up a lot. The “best” way to do this is a contentious topic around here.

Personally I recommend FromFile, which does what you’re after.

(Others prefer some include + using chicanery, which has a lot of boilerplate and suffers from a number of footguns. You can tell which side of this war I’m on.)

13 Likes
3

No, your scripts in runnable_file_?.jl should merely do using MyLib, and be executed with MyLib being the “active” project. I.e in order to run runnable_file_1.jl, you can

  • from the command line, if the current directory is the “top directory” of the project:
    shell> julia --project runnable_file_1.jl
  • from within a julia REPL:
    julia> ] # press `]` to enter Pkg mode
(@v1.8) pkg> activate .
(MyLib) pkg> # press <backspace> to exit Pkg mode
julia> include("runnable_file_1.jl")

In the standard way of doing things, MyLib.jl should definitely have include("SomeModule.jl"). What else it has depends on what you want to do. Assuming SomeModule exports a function foo:

  • if you do nothing:
    • MyLib can refer to foo as SomeModule.foo
    • runnable_file_1.jl can refer to foo as MyLib.SomeModule.foo
  • if MyLib does using .SomeModule (mind the . in front of the submodule name):
    • MyLib can refer to foo as simply foo
    • runnable_file_1.jl can refer to foo as MyLib.foo
  • if MyLib does export SomeModule
    • it does not change anything for the code written in MyLib.jl
    • runnable_file_1.jl can refer to SomeModule as simply SomeModule (instead of MyLib.SomeModule)

I’m not sure what “better” means, but I’ll say that a lot of packages only define the mandatory top-level module (MyLib in your example), but no submodule at all. Although most packages are sufficiently large that splitting their source code in multiple files, possibly grouped into subdirectories. Julia allows the two notions (file/directory structure on the one side, and module/submodule hierarchy on the other side) to be entirely decoupled, which allows package developers to adopt a lot of different structures.

Using FromFile is perfectly fine. Another way of doing things, relying only on standard Julia features and provided that you don’t absolutely need to insulate the various parts of your project in different namespaces, would look like:

Project.toml
src/
    MyLib.jl
    feature1.jl
    feature1/
        foo.jl
        far.jl
runnable_file_1.jl

with:

#file MyLib.jl
module MyLib
export a_user_facing_function # this export statement could have been placed somewhere else, for example right before the definition of `a_user_facing_function` in `feature1/foo.jl`

include("feature1.jl")

#file feature1.jl
include("feature1/foo.jl")
include("feature1/bar.jl")

#file feature1/foo.jl
a_user_facing_function(x) = 2*an_internal_function(x)
an_internal_function(x) = x + 1

#file runnable_file_1.jl
using MyLib

# can use a_user_facing_function directly because it was exported by MyLib
println(a_user_facing_function(42))
7 Likes
4

Another way is to start the runnable script with

using Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()

The third line is optional but means that dependencies will be automatically installed if they are missing, e.g. when running this the first time on another machine.

5 Likes
5

Interesting, FromFile and the specification it mimics sounds like at least a syntax that’s easier to follow than the status quo. Thanks for the link!

Just for curiosity - was there ever a proposal for anything that builds the module tree off of paths, rather than needing the module Foo within the included file? I’d personally be partial to something similar to rust - e.g. module SomeModule / export module SomeModule creates a module from file SomeModule.jl, and anything in a same-named folder could be used as submodules (if also declared). Kinda similar to Python, but without needing __init__.py and with easier reexporting

1 Like
6

Thanks for the thorough response. Your example is kinda sorta what I’m currently doing, but it just seemed very clunky to need to use include + export + module (with the same file name) to build project structure.

Julia allows the two notions (file/directory structure on the one side, and module/submodule hierarchy on the other side) to be entirely decoupled, which allows package developers to adopt a lot of different structures.

This is probably the context that I was missing, coming from languages that typically have them closely coupled. this page does sort mention that:

Files and file names are mostly unrelated to modules; modules are associated only with module expressions. One can have multiple files per module, and multiple modules per file.

But doesn’t talk about files/import again, and doesn’t clarify why it’s “mostly unrelated” and not completely unrelated.

7

Yes, while the currently available (or default) design patterns have served many Julia users well, I think they are a bit of a sore point for a sizeable chunk of other users. There are quite a few GH issues with a long & active debate on what exactly would be the best design moving forward. I am optimistic that __eventually__ there will be a resolution which will mean users write a lot fewer include(...)s, but I wouldn’t hold your breath :slight_smile:

1 Like
8

Well… I’m just glad to know I’m not the only one who struggles a bit :slightly_smiling_face: I’m just a bit surprised these answers and the “best practices” are a bit tricky to track down.

1 Like
9

For clarity (in case this is what you were thinking), FromFile doesn’t need a module Foo inside the include file.

Meanwhile as others have commented, the alternate approach to this usually involves creating just a single module for the whole project, and then just includeing everything into that one giant namespace. So this again doesn’t involve creating any extra modules inside files.

In terms of discussions: the main one on this topic is #4600.

3 Likes