Implicitly loaded modules in the future?

[jzr]

Does the content of utils.jl get executed twice in your proposed version, or is it cached somehow?

[patrick-kidger]

Only once; it’s cached.

[Skoffer]

It’s it optional, or everyone must follow it (i.e. current way become unavailable)?

[patrick-kidger]

I don’t think anyone is advocating removing the old way, if only for backward compatibility.

[StefanKarpinski]

Btw, you would not be allowed to modify foo and bar in another package from outside of the package without using eval.

[Raf]

My point is that a lot of the comparisons and sources of pain discussed in this thread are from moving from python. It appears that making that move from python easy for people is a major motivator for these changes.

It may not be yours, but you’re not OP here. OP specifically references python in the first post, as have others.

in which A.jl and B.jl implicitly depend upon utils.jl .

Also, is there a real package where this happens and is a problem? please link. Do you mean that capitalised A and B are modules?

[liuyxpp]

Indeed, with current include, it is so painful to read other’s package. I am always find myself locating where does this keyword/funciton/variable comes from by going through all source files. It really can drive someone crazy going this cycle over and over again!

[PetrKryslUCSD]

I wonder, have you seen this: Implicitly loaded modules in the future? - #72 by PetrKryslUCSD? It happens to mention your code. My point is that what is accomplished with FromFile macros could have been done with modules.

[PetrKryslUCSD]

@liuyxpp

Indeed, with current include , it is so painful to read other’s package.

Doesn’t that depend on the quality of the design of the package? Bad design is bad design, the language cannot prevent that.

[heliosdrm]

@patrick-kidger

The hope is to switch this out for a syntax looking something like:

# master.jl
import "A.jl": foo
import "B.jl": bar

# utils.jl
common = 1

# A.jl
import "utils.jl": common
foo = common + 2

# B.jl
import "utils.jl": common
bar = common + 3

Since import is meant to be an operation with modules, wouldn’t this be clearer if it required A.jl and B.jl to be written as “module files” (i.e., its contents being wrapped in module A ... end, etc.)?

That would mean typing a few more characters in each file, but it would make that new way of using import more understandable in my eyes: it would be more like a shorthand for include("A.jl") + import .A except for avoiding issues of multiple includes. Also, as soon as I see a file like A.jl in a project, I would clearly understand that foo belongs to the file-module A, so it won’t be writable by code from any other file.

This thread is long, and I have not looked thoroughly into all the answers or links to other posts or Github issues, so maybe the advantages or disadvantages of my suggestion have already been discussed elsewhere. If that’s the case, please don’t repeat the arguments; some pointers to relevant comments would be enough, thank you.

[patrick-kidger]

Yep, this is a minor point that’s come up once or twice. I don’t think there’s particularly strong feelings on this either way. (Perhaps apart from @PetrKryslUCSD?)

The module-less approach has less boilerplate, and the import statement can be understood to wrap a file into a module.

The module-full approach makes it harder to accidentally misuse any some_obj functionality defined outside the file, if it is unintentionally include-d, as plain some_obj usages would fail.

(Personally I favour the module-less approach. Less boilerplate is nice, and if import becomes the default approach then the same safety is attained. But I see the counterargument, especially during a transition period.)

[Orbots]

Not reading this entire thread, but linking my old thread as support for OP. Organization of multiple modules in same package

[PetrKryslUCSD]

@patrick-kidger

(Personally I favour the module-less approach. Less boilerplate is nice, and if import becomes the default approach then the same safety is attained. But I see the counterargument, especially during a transition period.)

I think the boilerplate is minimal, while the loss of legibility of the code/ability of the programmer to grok the code due to the missing module declaration may well defeat the purpose of this whole effort.

[MilesCranmer]

@PetrKryslUCSD thanks; indeed I did read your post as well as @gustaphe and @kristoffer.carlsson’s points about using submodules instead. I think it will be a bit more straightforward for everyone if you think of FromFile.jl as simply a macro on submodules, which it is (and the point of macros being to reduce unnecessary code).

I think the boilerplate is minimal, while the loss of legibility of the code/ability of the programmer to grok the code due to the missing module declaration may well defeat the purpose of this whole effort.

I think this is a bit of wishful thinking on programmers being as disciplined as yourself (your project looks very nicely organized!). For instance: me - I knew about the existence of submodules when I started my SymbolicRegression.jl project, but I did not use them. Why would I put every file in its own submodule when the include(...) is already enough? So that is what I did. But FromFile.jl makes it convenient and attractive enough to the point where I actually write down all the dependencies, so I actually do it! That is the main difference that I see for this efficient syntax, compared to manually writing out the submodules. [I don’t know if anybody has read “Atomic Habits” by James Clear - but this would essentially follow his point of making a desired habit “attractive and easy”]

Cheers,
Miles

[Skoffer]

Sorry, one more question. If it is optional, why not to use FromFile.jl or any other analogue to achieve necessary result? Is there anything in proposed approach/approaches which can’t be implemented in standalone package?

I mean, this is a good thing, that lots of ideas can be implemented as packages without being part of Base. If it can live outside of Base it seems that it’s better to stay this way. Just use package that you like and everyone is happy and those who support this idea and those who opposes it.

[Henrique_Becker]

For me it is useful to understand the main differences between the proposed syntax and the way I deal with organizing the Julia files. I include every file (with the files wrapped in modules) to the main package module and then making relative imports to them. The result seems similar to what you achieve, the main difference are:

1. Some boilerplate is removed (i.e., you do not need to wrap every file in a module).
2. The main package module will not have every submodule directly inside it (so it can be relatively imported by the other submodules).

I am not entirely sure if my approach is dependent on order or not, I do not remember anymore. If it is, then this would be probably the main difference, the DAG vs strict sequence is where more seems to be gained.

I like the current system, and my only problem with it is the lack of internal modularity. I would like the possibility of having a small set of “sub-packages” inside the package folder directory (only be visible to the main package). This way you could organize internal dependencies the exact same way you organize external ones. All the machinery would already be there, and it allows for DAGs already.

[lmiq]

@Henrique_Becker

I like the current system, and my only problem with it is the lack of internal modularity. I would like the possibility of having a small set of “sub-packages” inside the package folder directory (only be visible to the main package). This way you could organize internal dependencies the exact same way you organize external ones. All the machinery would already be there, and it allows for DAGs already.

+1 for this

Each package could have its own mini-registry. And open the corresponding subpackages to the world by putting them in the general registry. We would need to define which character would be used to separate the main package from its subpackages

] add BigPackage_SubPackage

using BigPackage_Subpackage

to install that one independently. Of course . is the character of dreams, but I fear that one is problematic.

I see this solutions as really a gain-gain. Not breaking. Allows complete modularity and allows this modularity to be exported to the world. And completely optional.

[lamorton]

Someone suggested an addition to the “Noteworthy differences” page. Here’s a draft that could be further wordsmithed. Comment on `include` pattern in Julia vs. Python by lamorton · Pull Request #41227 · JuliaLang/julia · GitHub

In particular, perhaps one might add something like: “To recreate Python’s module/file equivalence when structuring a package, do the following. (1) Wrap the contents of each file with a module MySubmodule, (2) export all the variable names from that module that should be made available in the main module, (3) do include("MySubmodule.jl"); import .MySubmodule within the main module.”

Or, maybe make a reference to “FromFile” or “PatModules”, if that is considered acceptable.

[aplavin]

Another datapoint: a very similar idea of “include-and-wrap-into-module” was suggested and implemented by @fonsp in the context of Pluto.jl: see include file containing structs · Issue #115 · fonsp/Pluto.jl · GitHub and around.

[liuyxpp]

I am not the package author, so it is difficult to figure out where does one function come from if it is defined in one of included files, such as

include("dir1/A.jl")
include("dir1/Ax.jl")

include("dir2/Bx.jl")
include("dir2/B.jl")  # some_func is defined in this file
include("dir2/By.jl")

include("dir3/C.jl")

# in file D.jl
f(x) = some_func(x)

Do you mean this is a bad way to organize codes? But I see a lot of prominent packages organized in this fashion, e.g. Optim.jl. When I first see some_func (I may visit this particular place by google searching), how can I tell it is defined in file “dir2/B.jl”?