Is there anything like Python's import or Rust's mod/use in Julia?

General Usage
,
1

Hello, everyone. I am asking a question on Julia’s code loading mechanism.

In Python, import some_module can be used for both importing some package in site packages and loading a module file in the same directory. However, in Julia, import or using can only load packages in the registry, but not load source files in the same directory. Julia do provides include for code loading, but include has the following drawbacks:

  • Unlike import, include brings everything in the same namespace
  • Unlike import, include won’t check whether the code has been loaded

This causes the “including multiple times” problems. For example, when I am writing code for the CAs of the pattern recognition course, I implemented a deep learning framework in Julia, and these are the source files in the project:

  • module.jl contains the definition and some basic operations of the Module type, which is the building block of a deep learning model
  • model.jl contains the definition of the Model type, which represents a deep learning model. It supports very limited autodiff based on Jacobian matrices
  • moduledef.jl contains definitions of several Modules, like Linear, ReLu, Sigmoid
  • train.jl contains the code to initialize and train a model

Both model.jl and moduledef.jl include module.jl for the definition of Module. And train.jl includes both model.jl and moduledef.jl. Since the include in Julia behaves just like #include in C, module.jl will be included twice. If the functions in those files are in the top level, they are just brought to the top level of train.jl and defined twice. If all the code are wrapped in a Julia module (say, with the same name of the source file), you will see Main.model.module and Main.moduledef.module. They are different Julia modules, but with exactly the same content!

import / using do provide mechanisms to resolve this. However, to use import or using, you have to:

  1. Create a subdiretory with its own Project.toml
  2. Place you source in the respective subdirectory
  3. In you project root, run ] dev subdirectory for each subdirectory

This is a lot of trouble, especially when the dependency between source files is complicated.

I have read Julia’s official documentation on code loading There’s no example of best practice of a project layout or how to split your code into multiple files.

This really imposes a lot of challenge when you are working with a large Julia project. If Julia supports C like #ifdef mechanism, or Python like import, or Rust like mod / use, this problem could have been elegantly solved. I have done some STFW, but I haven’t found any solutions.

1 Like
2

Many questions here, I won’t try to give a comprehensive answer (perhaps try with more focused questions in the future?).

Also, I’ll note there’s been many threads on these topics on these forums already.

The first thing I’ll say is: just create a package, it’s a great practice to make your deps explicit and will save you a lot of headache in the long run.

Use PkgTemplates.jl:

using PkgTemplates: generate
generate()  # starts interactive interface

There’s several packages providing the functionality you’re requesting. Personally I don’t find them useful, though. While include expressions (the equivalent of the #include preprocessor directive in C/C++) are awkward in theory, because of the need to order them correctly and keep them unique; in practice I don’t think that’s a problem at all:

  • any failures tend to be loud, use --warn-overwrite=yes --depwarn=yes, turned on by default during testing
  • the includes don’t need to be changed often while developing a package, at least in my experience
  • TBH lately I’ve started to prefer to just keep the entire package in a single file

Here’s my preferred style:

module MyPackage
    module SubModule1
        # ...
    end

    module SubModule2
        using ..SubModule1
        # ...
    end

    export public_name
    const public_name = ...
end

If you want to split the above into multiple files:

  • put each top-level submodule into a separate file
  • only use include at the top level, do not use it from submodules

There’s @static if etc. Or just the regular if.

1 Like
3

Just include at the top level only.

What is the reason to want includes anywhere else? This is a common discussion but I never understand it. You can import modules from anywhere in any other module.

Making packages is also super easy as @nsajko says, at some point of complexity its usually better to split out functionality anyway.

3 Likes
4

I’m not sure this solves exactly your problem, but: FromFile.jl is a package that implements somewhat python-like per-file module semantics.

1 Like
5

I don’t think Julia provides very good tools for this. and thus I agree with your assessment that there are not many good examples to point at.

I know this is a bit of an unproductive statement, but I’m just saying it to empathize; I don’t think your confusion is your fault.

1 Like
6

modernjuliaworkflows.org has some best practices, though I don’t see anything specifically about how to split code into files within a package, or about submodules. They do discuss using local packages though: Writing your code

3 Likes