What is the best workflow when juggling with many modules and files?

Hi, I’m a bit unsure about what would be the best way to import a module with using. I understand, looking throught past julia topics, that if you want to import a local module, let’s say Users.jl, you should do include(Users.jl) and then you can do using .Users to get it back and indeed it works. However I saw, again in a past julia topic, that if “you wish to access the features of a Julia package within your code, you must either call import or using (as opposed to include)”. I indeed only want to use features of module however if I don’t include it beforehand I have an error saying that Users is undefined. I tried to push! to LOAD_PATH the path of the folder in which the Users.jl module is defined but it did not work and anyway as I rade it doesn’t seem like it’s an ideal solution… In my real case scenario I have a lot of modules, each one being defined inside different folders. LOAD_PATH seemed initially an ideal way as I thought I wouldn’t have to include one by one all my modules and would just be able to push to it the top level folder containing every subfolders, and then be able to access them every where in the code by doing using myModule. But did not work… Right now the code is working but if you have any idea of what the ideal way to proceed would be, I would really appreciate it.
This is my code structure :
code_structure

This is my code :

#main.jl
cd(@__DIR__)
function main()
    include(joinpath("app", "App.jl"))
    Base.eval(Main, :(using Revise))
end; 
main()
using App.Users
using Revise

#App.jl
module App
function main()
    include(joinpath("config", "config.jl"))
end; main()
end

#config.jl
include(joinpath("..", "resources","users","Users.jl"))
#This is what I tried with the LOAD_PATH variable
# const ROOT_PATH = pwd()
# LOAD_PATH = ["@", "@v#.#", "@stdlib", ROOT_PATH]
# appPath = joinpath(ROOT_PATH,"app")
# resourcesPath = joinpath(appPath,"resources")
# usersPath = joinpath(resourcesPath,"users")
# LOAD_PATH = ["@", "@v#.#", "@stdlib",ROOT_PATH, appPath, resourcesPath, usersPath]
#println(LOAD_PATH)
include(joinpath("..","services","service1.jl"))
using .Users

#Users.jl
module Users
export User
mutable struct User 
  id::Integer 
  name :: String
end
end

#service1.jl
using .Users
myUser = User(123,"agathe")
println(myUser)

There is different using a bit every where in the code to help me understand how to import module depending on where I am but the main goal is to use in services files, modules defined in the resources folder (many modules in real case scenario) , and to include this services files (many files in real case scenario) into the config.jl file. Furthermore this config.jl file is called from an App.jl file via include. By mimetism (from other projects) I turned this App.jl into a module that is called from a main.jl file but I think I don’t really get the advantages of doing so… what do I get from my App being a module ? Finally last small precision, I run this code from julia REPL by writing include("main.jl").
My apologies in advance if my questions are unclear or if I put too many questions in one topic, don’t hesitate to tell me, I will do better next time. Thank you !

I don’t think that you can include a file that defines a module inside a function. Modules are top-level construct.

With Julia 1.5.3 the code you posted did not work. This works:

#main.jl
cd(@__DIR__)
using Revise
include(joinpath("app", "App.jl"))

using .App
using .App.Users

Perhaps the bigger picture question is: do you really need that many modules?

One drawback of having lots of modules is precisely the issue that you are running into now: one has to issue lots of import and export statements.

Even large packages are often written as a single module (not always). Unless there are name conflicts, dividing the code into separate modules does not really accomplish anything. Unless you plan to reuse a module somewhere else, of course. But then it should be a package.

5 Likes

When not using modules for code reuse (i.e. create module X that is used by program A and B) I find that modules are best used to avoid name collision.

That said with Julia you can happily have multiple methods with the same name as long as they have different parameters. If you are not running into name/parameter collisions then you probably don’t need to put everything in modules. However is different people are responsible for different “modules” or directories then maybe modules will help just to avoid accidental name collision.

Regardless of if you do modules or not the way I tend to organize my code is:

src
     module1
        module11
            _module.jl
            m11_file1.jl
            m11_file2.jl
        _module.jl
        m1_file1.jl
        m1_file2.jl
     module2
         _module.jl
         m1_file1.jl
         m2_file2.jl
    file1.jl
    file2.jl

The _module.jl files really only contain the includes for all the other Julia files in that directory and include the _module.jl file(s) in the lower directories. Sometimes they define some “high level” objects that the other julia files in that directory need. This way if I add a file to a module the file to add the include too is “near” it.

If I am worrying about name collision, I can then just wrap the includes for a particular _module.jl file inside a module block and possible put a using statement after it i.e.:

module Module1

include("module11/_module.jl")
include("m1_file1.jl")
include("m2_file2.jl")

export find, path

end

using .Module1

That way when I include Module1/_module.jl the module is defined in it’s own namespace and the “public” methods are exported automatically.

Correction: it is possible to include inside a function. This may define a module, but it does that in the global scope. So I don’t think that is what you wanted.

I am not sure I understand when you say “It does that in the global scope”, which scope are you talking about ? The module Main or the module in which the function is called ? And either way I am having a hard time understanding why that wouldn’t be something I want, what does it imply ?

Oh yes sorry I forgot to mention it but I am using julia 1.3.1

Good question…Maybe I indeed don’t need that many module. I thought I did as I thought it would be lighter to have small modules and import only the small module I would need in a particular file. Plus each module right now defines a specific object with a struct defining it and functions to access it, that’s why it made more sense to me to have one module for one object. But maybe I just don’t really understand how to use modules yet… Would you suggest that I should have different files, one for each object (my current modules) and then include those files in one big module, which would mean that I would have to import everything everytime I need something from one of those files, but maybe that’s not really an issue. And I don’t think I will have name conflict as I want each module to define a different object. The struct and functions to handle that struct will thus have different names. However I am not sur what you mean by “unless you plan to reuse a module somewhere else”. I indeed plan on reusing the code as I want to use the structs and functions created in my modules in different places. But if you mean in a different project than mine then no I don’t plan on reusing those modules in another project.

I don’t understand why your Module1 would look like that. If “The _module.jl files really only contain the includes for all the other Julia files in that directory and include the _module.jl file(s) in the lower directories” then I would have thought to have something like that :


#_module.jl (the one that module1 sees)
include("m1_file1.jl")
include("m2_file2.jl")
include("_module11/_module.jl")

#module1
module Module1
include("_module.jl")
export ....
end

#file1.jl
using .Module1

What do you have in you Module1/_module.jl ? And what does it imply that “the module is defined in it’s own namespace”

That depends on what you call “every time I need”. If that is within the package you are developing, just don’t use the multiple-modules and you do not need to import anything, the usual layout is:

module MyPkg
  export MyStruct, myfunc
  include("./MyStruct.jl")
  include("./myfunc.jl")
end

where in MyStruct.jl you just define the structure (without any module) and in myfunc.jl the function, or functions. How to split the code into files is more or less at will, that depends on how you think the code is better organized (I use to put one function in each file, but that is not the most common pattern here, apparently).

If, on the other side, you want, for example, that MyStruct is available to other packages, or independently for being used without myfunc, you can define a package only for that. For example:

module MyStructsPkg
  export MyStruct
  include("./MyStruct.jl")
end

Then, the previous one would be

module MyPkg
  using MyStructsPkg
  export myfunc
  include("./myfunc.jl")
end

and you could have another package that uses MyStructsPkg and does something else, or you could just load MyStructsPkg in the REPL and use those structs directly. That is pretty common, actually, there are many packages that define custom structures (Data Types). In general they define the functions that allow the most common operations that one user would want to do with that data type.

But that splitting in packages is only interesting if your new type is really interesting outside the context of the package you are developing.

When I organize things a module only calls sibling or children, it doesn’t access children of siblings (directly). Sometimes a sibling might “re-export” a method or structure of it’s child, but that it. This is just the way I end up organizing, it might be because of my Java background, I don’t really know. If a module needs to be accessed by an uncle or grand parent then I usually move it up in the hierarchy.

Any function/structures defined in a module are in their own namespace, meaning their function/structure names won’t conflict with the functions/structures in other modules. So Module1 and Module2 can both define their own version of a structure Foo without any issues. If Foo is exported by Module1 and Module2 then you might start having issues, but as long as Foo is only used internally nothing bad happens.

As the manual states:

So if you have function main and that function includes a source file, whenever you call that function, the source file will be evaluated in the global scope. It seems to me that what you wanted was to run the application from within main, not to prepare the source code for execution. (I may be wrong, of course.)

The way I restructured your code above, the source code was evaluated once, making the modules available for use.

All right thank you for all your advices. Lots to think about…