Ok, let’s make this more concrete, hopefully it will become clearer this way.
I have three modules in three separate files:
Base.jl:
module Base
export do_stuff
function do_stuff end
end
Support.jl:
module Support
# I would like to do this, but I can't:
# import Base
# so this is what I do instead:
import ..Base
Base.do_stuff(x::Float64) = ...
Base.do_stuff(x::Int)=...
end
Macro.jl (can’t remember why I called it D before):
module Macro
export @mymacro
# I would like to do this, but I can't:
# using Base
# so this is what I do instead:
using ..Base
function helper(x)
# this needs to call a method of Base.do_stuff
do_stuff(x)
end
macro mymacro(...)
...
# does complicated stuff, generated code calls helper on user-provided type
end
end
In order to make this work I need a fourth file:
Global.jl:
module Global
include("Base.jl")
include("Support.jl")
include("Macro.jl")
end
Edit: I forgot to add, that all of these are part of a package.
Yes, as I said, the top-level file in a Julia package is analogous to a “makefile” in another language. It’s where you list all of the files that make up your package. I guess I’m used to makefiles from other languages so this doesn’t seem so bad?
If it actually was some kind of makefile equivalent I wouldn’t even mind so much, as a makefile represents the dependencies within a project. However, this is just a flat list of files with essentially no information content and that could be trivially generated.
I think the problem is that Julia’s module system tries to be completely orthogonal to the file system but because that would be too inconvenient ends up in a weird place that’s neither nor. If using/import did really only handle administration of available namespaces and their contents then a separate (potentially flexible and powerful) system would have to exist that tells the compiler which files to load the module from. As it is we have a hybrid, though, where using/import a) do stuff with modules that the compiler already knows about, b) search through the file system for files that might contain an unknown but requested module and c) negotiate with the package system to find an entire directory structure that represents the requested module. Unfortunately all of that is context dependent as well, so stuff that works in one context (e.g. plain directory) does not work in another (package directory).
Anyway, none of that is really terrible and workarounds exist. I guess even a really well-designed language such as Julia will end up with a wart or two (see also e.g. issues #40717 or #269). However, I think a simple local import would already solve 90% of the issues people have.
It could be generated from what? By assuming there is a one-to-one correspondence between files and submodules? That isn’t typical for Julia packages. Or just assuming every .jl file in the package src tree should be included? That seems a bit dicey.
Or are you just referring to the old suggestion, mentioned above, that using .Foo should search the current directory for Foo.jl? That’s possible, but it wouldn’t eliminate most usage of include, because most packages seem to use multiple files per module.
In first approximation a list of all .jl files whose names begin with capital letters would probably come quite close, but that’s besides the point. You are right, of course, that there are many different scenarios where this list could not actually be generated. That doesn’t change my main point, however, that (as opposed to a makefile) this file contains very little useful information and its only purpose really is to compensate for a weakness of the module system.
I agree that the module system tries to be orthogonal to the file system, but I’m not sure I completely understand why you say Julia does not achieve this goal.
So if the module system is meant to be completely orthogonal to the filesystem, then there needs to exist both a list of modules (defining where names are to be looked for) and a list of source files (defining where the source code lives in the filesystem).
Currently, this list of source files is built from two sources of information:
the Pkg system, and
the set of include calls.
In that regard, a list of included files can surely be considered to contain useful information. Or did I misunderstand your point?
Include has a very different function though, that only partially (and I would argue, accidentally) overlaps with the module system. Julia’s include works more or less like C’s, so it has the same strengths and weaknesses. It’s super flexible because it allows you to chop up your code however you like. At the same time it’s very stupid, however, and simply copies and pastes the code without much in terms of sanity or dependency checking.
I think the module system should be completely independent of include.
You can define a complex hierarchy of dozens of modules in a single package without having to call include once. Or you can have only one module in your package and split it in dozens of files with include (like the majority of Julia packages). If that isn’t independence, what is?
Maybe I’m being dense, but I don’t see how you would do that unless either all of these modules exist in a single file or none of them refer to any of the others.
This isn’t unusual among the wider languages, and Julia didn’t come up with it.
The only ways your example’s import ..Base (putting aside the dangerous name conflict with Base for now) would work without Global.jl specifying the modules structure is if files encapsulated modules, making the surrounding module ... end expressions entirely redundant in your example. It works for Python because files-as-modules are designed to heavily import from each other. Heavy use of from OtherMod import * can mimic include-ing a dozen files into 1 Julia module, but the imported names still come from a dozen other modules and that difference can matter.
To make multiple files evaluate into 1 module from the start, we must be able to evaluate files into a global scope (include) separate from importing modules. At that point, we might still treat files as modules, we just include some into others that we import. But how obvious will that be? Say Macro.jl gets long enough we split the helper functions into Helpers.jl. We can’t tell just from the file names how they’re related, we have to open both of them to find out Macro.jl runs include("Helpers.jl"). If we had >10 files, where do we even start? Note that Python has ways to evaluate files (and this is crucial for IDEs running scripts because import does not reload code on its own), and it’s almost never used alongside import this way because of how much of a nightmare it is to handle.
The module expression becomes useful because files can no longer tell us what should or shouldn’t be imported. 1 conventionally named file per directory gives us a starting point for the code’s module structure (Python packages’ directories also have this). That pretty much gets us to the status quo.
I’m not entirely sure which point you are trying to make (maybe because my experience with Python is quite limited). In any case, I think it might be useful to be a bit more precise WRT terminology (I certainly wasn’t).
You are right, of course, that there is no file ↔ module correspondence in Julia and that, therefore, strictly speaking there can be no straightforward way to import local modules. There is, however, under certain conditions a correspondence file ↔ package. By adding a directory containing single-file packages to the load path you can effectively import modules corresponding to files. Unfortunately that doesn’t work as soon as you are in a project environment, since then only explicit dependencies can be loaded.
So, again trying to keep the terminology straight, I think what is missing in Julia is the ability to add the current (or any other “local”) directory as a package directory within a project environment. If that was possible, then - going back to my example - Base.jl could be loaded by a simple using or import.
You don’t need to know why - that’s not important. What matters is that someone wants to do it. If there is no way to do it, then that’s fine. I will go tell them it cannot be done.
I don’t know why they didn’t just do the Rust/Python thing which works really well.
I really like your response, it helped to clarify a few things in my own mind. It’s really bizarre that using/include can both load from file and perform operations on the current (in memory) module system.
If Julia wanted to go the C++ route and have includes everywhere to explicitly load code from disk (something that Python specifically tried to avoid) then using and include should have only been for module manipulation and include should have worked when targeted at a package directory as well as just a single file.
The other big problem with it is you can include more than once. Which is a disaster, and experience has demonstrated to me that devs will do this without realizing the consequences of what they are doing.
I think in parts it’s a UI issue. If there was a clear separation between commands that operate on modules, files and packages respectively the distinction would be immediately obvious and it would be clear to everyone that Julia modules do not behave like the Python (or whatever) equivalent. By using using (heh) for everything the distinction becomes muddled and people get confused.