Implicitly loaded modules in the future?

Ahh, it finally clicked for me! Thanks for your patience (and the patience of the other experts in this thread). It took me an embarrassingly long time to stop conflating files and namespaces subconsciously. :grimacing:

Tracing the dependency DAG is a matter of tooling, whether the module is all in one file or not - just needs smarter tooling in the latter case. I also changed my mind on the value of using submodules as a method to signpost the dependency flow. Either there is redundancy (keeping the name of something the same in both the inner and outer module and needing to say so on both sides of the boundary using explicit export/import statements) or confusion (if the name gets changed going across the boundary).

The guarantee that include gives when reading a package (assuming that the package isn’t triggering redefinition warnings) is that a name means the same thing in any file – because they’re all in the same namespace. The warning mechanism is the key, though.

4 Likes

In my understanding this covers only some of the things modules can be. Here’s a list of what modules in Java 9 aim for:

According to JSR 376, the key goals of modularizing the Java SE platform are

Reliable configuration—Modularity provides mechanisms for explicitly declaring dependencies between modules in a manner that’s recognized both at compile time and execution time. The system can walk through these dependencies to determine the subset of all modules required to support your app.
Strong encapsulation—The packages in a module are accessible to other modules only if the module explicitly exports them. Even then, another module cannot use those packages unless it explicitly states that it requires the other module’s capabilities. This improves platform security because fewer classes are accessible to potential attackers. You may find that considering modularity helps you come up with cleaner, more logical designs.
Scalable Java platform—Previously, the Java platform was a monolith consisting of a massive number of packages, making it challenging to develop, maintain and evolve. It couldn’t be easily subsetted. The platform is now modularized into 95 modules (this number might change as Java evolves). You can create custom runtimes consisting of only modules you need for your apps or the devices you’re targeting. For example, if a device does not support GUIs, you could create a runtime that does not include the GUI modules, significantly reducing the runtime’s size.
Greater platform integrity—Before Java 9, it was possible to use many classes in the platform that were not meant for use by an app’s classes. With strong encapsulation, these internal APIs are truly encapsulated and hidden from apps using the platform. This can make migrating legacy code to modularized Java 9 problematic if your code depends on internal APIs.
Improved performance—The JVM uses various optimization techniques to improve application performance. JSR 376 indicates that these techniques are more effective when it’s known in advance that required types are located only in specific modules.

From https://www.oracle.com/corporate/features/understanding-java-9-modules.html

Some of these are not relevant, but it would be beneficial to be able to understand dependencies between modules better. It would also be useful to be able to create a module, add packages for that, and have some ensurance that the packages you add don’t mess up with code that you didn’t want changed.

Even better, I use parametric include statements :sweat_smile:. I (ab)use a file containing a declaration of a struct with default values (courtesy Parameters.jl) as a parameter file for my simulation and the location of that file can be given on the command line. I am aware that in most contexts this would be a terrible idea, but for this particular use case I find it quite handy.

Edit: Oh and I forgot, I do the same thing with “scenario” files, which change some aspects of the simulation and are loaded on demand. Basically like a plugin system, but really simplistic.

2 Likes

@mhinsch ModelParameters.jl is a cleaner way to do this kind of thing. You can just use a csv :wink:

1 Like

7 posts were split to a new topic: Topic title change to the module thread

A post was merged into an existing topic: Topic title change to the module thread

Nice, I didn’t know about that one, I’ll look into it. In any case I could also just read parameters from a json file or something like that.

The one advantage of doing it the way I’m doing it - besides being dead simple - is that I have a single point of definition for parameters, their default values and their documentation (I’m using the doc strings of the struct members to generate help text for the command line) that immediately results in a runnable model. Plus, no additional syntax for the parameter file.

ModelParameters is for working with a single point of definition for those things, but by automating all the struct reconstruction instead of including code. Parameters, labels. priors, bounds, whatever else you need can be on rows of a csv and attached to model parameters.

There’s no syntax except defining the (potentially nested) structs with Param wrappers where the parameters should go. You can use Parameters.jl or Base.@kw_def to set those defaults, similar to how you are now.

2 Likes

It’s not always that someone can’t think outside the python box. It can be that they understand both Julia and Python to some degree and see some language-feature that one has and the other doesn’t. Or even if both languages have the feature, they see an advantage in borrowing a bit from cultural practices.

I learned Python after Julia and have far more experience in Julia. But, after having significant experience in Python, I started more often encapsulating code within a package to avoid that “A.jl and B.jl implicitly depend upon utils.jl”. I’m using Julia’s existing module system at the moment. For some projects, most of the code in each file is used only within the file. So, I tend to wrap the code a module. I often don’t even import symbols from these modules, but rather just the module name, and then fully qualify the symbol at the call site. Knowing where stuff comes from helps me remember how I organized the code. And making the dependencies explicit will make the code more maintainable as it scales up.

There are many Julia practices that I try to introduce in the Python code. And a few in the other direction. I should add that I am not a Python fan-person. I absolutely hate writing scientific software in Python, because it is fundamentally so ill-suited for that task.

EDIT: I also understand the desire to push back against extraordinarily-difficult-to-defend opinions, like: Julia doesn’t have a proper module system, or Julia was designed to be used only from Jupyter notebooks.

5 Likes

Yeah I wasn’t suggesting it was a limitation of python people not thinking outside the box, more that learning a language gives you a tendency to look for familiar structures - and as you say sometimes that may be a good thing. I know I avoid modules even when I reach the complexity where they would help.

Personally I wrote more Ruby, C++ (and ugggh PHP) than Python, and that experience makes Julias include seem normal and obvious, I don’t think about doing any of the things people are worried about here.

5 Likes

C++ is going to have something better than include (see for example: c++20 - What exactly are C++ modules? - Stack Overflow)

Julia has this already (with a number of additional features and improvements): packages.

8 Likes

Question of this thread (if I understood OP) is probably:
Why Julia doesn’t solve some problems (for example isolation and de-duplication) at module level?

In other words: What is the benefit of modules which doesn’t solve that automatically and require burden with include mechanism?

I am not trying to criticize Julia’s paradigm! I am just curious. It could be nice and useful to see ideas behind language design.

What I wanted to say with my reaction to Raf is that C++ designers (and not only them) probably don’t like include machinery very much.

1 Like

I followed this thread with interest, but one thing that seems to stay a bit out of focus is the differences in philosophy/design between Julia and (say) Python when it comes to packaging and reuse. This influences the way modules work in Julia and perhaps (partly) causes the perceived mismatches with other languages.

  • In Julia the package is the main unit of reuse and distribution, as it contains all necessary details on dependencies for a set of source files. Plus packages have explicit support for easy deployment via e.g. git repositories and management using Pkg.
  • Even though Julia packages are usually structured in modules, using a Julia module by itself is not really a supported goal (you need to go through a package instead) and it shows in the way they are implemented. For example, a module might be split over multiple source files and there is no general way to handle such a module by itself except by using the package the module is contained in.
  • In contrast in Python any .py file comprises a module. And as such it (implicitly) list its dependencies through the relevant import statements contained in it. So a Python module can usually be reused without too much trouble and is much more of a standalone unit than a Julia module. For example, this helps in writing test code for individual modules as you simply import the module to test in a driver script and don’t need to worry about the package the module is part of. The same goes for a Python package, which is not much more than a directory of files, leading to a set of nicely namespaced modules.
  • But support for module/package deployment is not as tightly integrated in the Python ecosystem, even though there are things like PyPI, setuptools and pip to handle packaging and deployment. All in all, handling dependencies is quite a bit more involved and requires more manual labour in Python than in Julia.

At least, this is my 10,000 feet view, based on limited experience with Julia (but much more with Python).

3 Likes

This is not true. The general non-package way to use a module defined in one or more files is to do include("file-with-mymodule.jl"); import .MyModule, where "file-with-mymodule.jl" is the file that contains module MyModule ... (and which possibly includes other files).

6 Likes

But it seems you have no guarantee in general that the included module file will pull in all necessary dependencies? As dependencies are quite often listed at the top-level package .jl file and not in the module file, with individual parts making up the package include()d below all the dependencies. I.e. structured like

module M

import ...
using ...

include("file1.jl")
include("file2.jl")
...

end

Including file1.jl as you suggest will most likely not work.

Yes it will, as long as file1.jl defines a module (i.e. it is of the form module ... end). Any import and using statements outside of the module statement (in the enclosing scope) don’t affect it.

8 Likes

Okay, but this nicely highlights the semantic differences of a “module” in Julia and one in Python. Even if file1.jl defines a module and lists all dependencies it can include() dozens of other source files which individually will not list their dependencies, and cannot easily be used by themselves. So indeed a Julia “module” will be usable outside of its enclosing package the level of use in general is coarser than in (say) Python where every source .py file is guaranteed to be importable by itself (and also each make up a module). I guess the main point I was trying to make was that in general in Julia a single .jl file does not equal a Julia module and is not expected to be easily usable by itself (yes, you can include() it, but that might require dependencies to be manually specified as well).

Yes, Julia is not Python—believe me, we know :wink: . But saying “files ≠ modules” in Julia is different from saying that “using a Julia module by itself” (not as a package) is not supported.

I think people here are generally open to ways that make it easier to import modules from files without having an explicit include, e.g. by import "Foo.jl" or some other syntax. But I think it’s unlikely that Julia will ever require each file to define a new namespace, ala Python.

12 Likes

This really captures where I find these threads lose me – the threads oscillate between two requests, one of which seems great and one of which seems not great:

  1. Great: It should be easier to treat a file like a module that has a sealed namespace so that importing it makes all introduced names explicit.
  2. Not Great: It should be impossible to break up a single module into multiple files.
14 Likes