Implicitly loaded modules in the future?

I find the modules implementation in Julia the most annoying “feature” of the language.

For large projects it makes sense that developers would want to organise their code in separate modules (not packages), each one in a separate directory with a number of files. But, in Julia the executed code should be formed so that it includes each module once and one after the other in the proper order of usage. This is like manually joining together all the files forming the code in the right order and therefore the concept of modules loses its meaning. Are there any plans for the future to build a proper module system like that present in other languages e.g. Python or Java?

1 Like

The order of modules don’t matter if they are independent. If they are very interdependent, then why are they modules? If they are interdependent, don’t you need to specify the direction of the dependencies?

9 Likes

They can have dependencies. E.g. I may have a Mesh module which does some geometry work and then a Flow module which uses the mesh for solving the flow equations using various methods stored in separate modules which may need e.g, a Thermodynamics module and a Flow Properties module which needs to compute gradients and therefore it also needs the Mesh module etc.

In a large code there are all kinds of possible dependencies that cannot be done in a way that somehow the modules are loaded one after the other in some particular sequence that includes them once and yet the result actually works correctly.

I think this is the same issue discussed here

2 Likes

I am not surprised to see that other people are also frustrated by the modules in Julia. I guess that if the primary purpose of Julia was to be used for Jupyter notebooks and relatively small pieces of code which make heavy use of external packages then the module system is not important. But for a large scale scientific/engineering project things become very difficult with this implementation of modular programming.

I wouldn’t say that many people are frustrated with modules. One thread doesn’t prove it.
I have used multiple modules in many (most of) my packages, and never with any ill effects or frustration.

14 Likes

Let’s say then that “some” people find it hard to cope with modules!

1 Like

Rereading that thread I don’t quite understand the situation.

Can you lay out a simple concrete scenario to discuss?

4 Likes

What I understand regarding the way Julia works is that it needs to have a single file with all the code inside and in the proper sequence (so that some piece of code does not e.g. uses types not defined yet but only later on) and then it executes it line after line until the end.

So, if you use multiple files you need to include them to some main file and run this file. If you have modules again you need to include their source code manually so that they are only included once and in the right order. As a result, modules are not giving you any value like in other programming languages because you need to manually take care of their code so that at the end you create this large file with all the code for Julia to execute. And let’s not get into the whole dot notation for Julia to locate the modules with the “using” statements!

It seems that the issues you raise were addressed in this thread

that you opened some years ago. There doesn’t seem to be any new complaints in this iteration.

Was the advice in that thread not helpful?

4 Likes

Apparently not! I’ve been waiting to see some improvements to the language ever since but modules remain the same.

Okay, but the feedback in that thread was basically “don’t use include that way”. It seemed like your questions were addressed, but I don’t know how the advice fell short.

8 Likes

If you are having trouble with modules, but other people manage apparently just fine, perhaps you are not using them in the right way?

4 Likes

If I convert my modules to packages then I would have no problems but is that the proper way to arrange a code when these packages are only meaningful as parts of a single and specific application?

1 Like

If you can convert the modules to individual packages, there should be no issues with interdependencies. In other words, such modules can be included and used in any order.

On the other hand, if the modules depend on functionality implemented in other modules, then either make your code one monolithic file without submodules (everybody lives in the same house - module), or separate into modules, but then the order in which the files are included to be compiled matters. Obviously, circular dependencies are not good. :wink:

4 Likes

I should note that the PatModules.jl thread spawned https://github.com/Roger-luo/FromFile.jl and https://github.com/JuliaLang/julia/issues/39235. The former can be used now, and the latter looks like it has enough traction to happen at some point.

Edit: see also https://github.com/JuliaLang/julia/issues/4600.

8 Likes

I don’t share this general frustration with modules, but one very specific thing that is probably related to what the OP describes as frustrating is this:

julia> module A
         f(x::M) = x.m
         struct M
           m
         end
       end
ERROR: UndefVarError: M not defined
Stacktrace:
 [1] top-level scope
   @ REPL[1]:2

Is it obvious why this does not work? (meaning, why the order matters there?).

Answering part of my question:

julia> module A
         f(x::Int) = x
         f(x::Int) = 2*x
       end
WARNING: replacing module A.
Main.A

julia> A.f(1)
2

But, then, couldn’t this simply throw an error? What are the situations in which good code requires that these two behaviours are as they are?

1 Like

Isn’t your example intrinsic to how Julia handles forward references given that it’s not a statically compiled language?

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.1640 (2020-12-02)
 _/ |\__'_|_|_|\__'_|  |  Commit 9528ac2785* (188 days old master)
|__/                   |

julia> f(x::T) = 1
ERROR: UndefVarError: T not defined
Stacktrace:
 [1] top-level scope
   @ REPL[1]:1
3 Likes

This behaviour is the result of Julia processing the file line by line like executing one-line commands.

Well, yes, but is anything in the module parsed before the “end” is returned? Is it possible to interact “inside” a module, such that that can’t be handled?

Having the things out of order, while sometimes practical, is certainly bad style. But having multiple conflicting definitions of the same method inside the module seems to be useless, and very bad style anyway. Are there “good” uses of that?

3 Likes