Problems with signature changes of imported local modules on include

[nsajko]

I agree, it’s a bit awkard compared to other languages (except C and C++). I’m not sure what is the opinion on the status quo by the authors of Julia, but there’s lots of historic discussion on the Julia Github, for example this issue.

Actually, I wasn’t completely accurate above, you could use the load path mechanism to be able to load modules located in a specified directory (or current directory). Then you wouldn’t need multiple packages, but personally I think you’d be better off with creating packages in the long term.

The good thing is that creating packages is really easy with PkgTemplates, and using them is also easy if you either:

1. use them only privately (in that case just dev or add them from the Pkg REPL)
2. publish them on the General registry

In case you want to share your packages with others, but you can’t publish them to the General registry, you’d have to create your own registry, which is presumably a bit more of a hassle.

[nsajko]

@hijit

Simultaneously I don’t want to achieve this by making the presence of any one of these modules necessary to use/import any of others.

I think this is the main thing that’s currently missing in the Julia module system, as long as you’re not willing to use packages. However I’d say it stops being an important issue for those who create packages, especially as a package can have many (nested) modules.

[nsajko]

@nsajko

So you want to be able to use the modules independently from each other? Then you need to organize them into multiple packages.

To be clear, you also always have the option of putting several modules into the same package, and then importing the package when you want to use some module contained by it. It might be a bit hackish if the modules are not related, but it can also be convenient.

[mkitti]

@hijit

include("MyModel.jl")

Including a file like this should occur exactly once in only one module. You should not be including it multiple times in multiple modules like this.

I’m confused because you do not actually declare moduleA or moduleB anywhere.

Let’s start from scratch.

Let’s construct a proper package.

$ tree
.
├── MyModel
│   ├── Project.toml
│   └── src
│       └── MyModel.jl
└── MyPackage
    ├── Manifest.toml
    ├── Project.toml
    ├── config.toml
    └── src
        ├── ModuleA.jl
        ├── ModuleB.jl
        └── MyPackage.jl

4 directories, 8 files



$ find . -print   -exec cat {} \;   -printf "\n"
.
cat: .: Is a directory
./MyPackage
cat: ./MyPackage: Is a directory
./MyPackage/Project.toml
name = "MyPackage"
uuid = "331a3bf3-8bf7-4b5c-8fe0-a68622373184"
authors = ["root "]
version = "0.1.0"

[deps]
MyModel = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

./MyPackage/src
cat: ./MyPackage/src: Is a directory
./MyPackage/src/MyPackage.jl
module MyPackage

import TOML
using MyModel: Model, getdata
# I recommend Preferences.jl here
const configs = TOML.parsefile(
    joinpath(@__DIR__, "..", "config.toml")
)
const runtimevars = configs["runtimevars"]
const models = Dict{String, Model}()

isentrypoint() = abspath(PROGRAM_FILE) == @__FILE__

for var in runtimevars
    models[var] = Model(var)
end

for var in runtimevars
    # We might confuse run time and compile time
    # by changing model in between includes
    # global model = Model(var)
    if var == "Evaluate in ModuleA."
        include("ModuleA.jl")
    elseif var == "Evaluate in ModuleB."
        include("ModuleB.jl")
    end
end

end # module MyPackage

./MyPackage/src/ModuleA.jl
module A
    using ..MyPackage: Model, getdata, isentrypoint, models
    processdataA(model::Model) = println(getdata(model))

    function __init__()
        if isentrypoint()
            println(getdata(models["Evaluate in ModuleA."]))
            println("We are evaluating in ModuleA.")
        end
    end
end


./MyPackage/src/ModuleB.jl
module B
   using MyModel: Model, getdata
   using ..MyPackage: models, isentrypoint
   processdataB(model::Model) = println(getdata(model))

   function __init__()
        if isentrypoint()
            processdataB(models["Evaluate in ModuleB."])
            println("We are evaluating in ModuleB.")
        end
    end
end

./MyPackage/Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.3"
manifest_format = "2.0"
project_hash = "a583b77fe794703b105dd8a19da10f8b2723c0b5"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.MyModel]]
path = "../MyModel"
uuid = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
version = "0.1.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

./MyPackage/config.toml
runtimevars=["Evaluate in ModuleA.", "Evaluate in ModuleB."]

./MyModel
cat: ./MyModel: Is a directory
./MyModel/Project.toml
name = "MyModel"
uuid = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
authors = ["root "]
version = "0.1.0"

./MyModel/src
cat: ./MyModel/src: Is a directory
./MyModel/src/MyModel.jl
module MyModel

struct Model
data::String
end
root@localhost:~/hijit# find . -print   -exec cat {} \;   -printf "\n"
.
cat: .: Is a directory
./MyPackage
cat: ./MyPackage: Is a directory
./MyPackage/Project.toml
name = "MyPackage"
uuid = "331a3bf3-8bf7-4b5c-8fe0-a68622373184"
authors = ["root "]
version = "0.1.0"

[deps]
MyModel = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

./MyPackage/src
cat: ./MyPackage/src: Is a directory
./MyPackage/src/MyPackage.jl
module MyPackage

import TOML
using MyModel: Model, getdata
# I recommend Preferences.jl here
const configs = TOML.parsefile(
    joinpath(@__DIR__, "..", "config.toml")
)
const runtimevars = configs["runtimevars"]
const models = Dict{String, Model}()

isentrypoint() = abspath(PROGRAM_FILE) == @__FILE__

for var in runtimevars
    models[var] = Model(var)
end

for var in runtimevars
    # We might confuse run time and compile time
    # by changing model in between includes
    # global model = Model(var)
    if var == "Evaluate in ModuleA."
        include("ModuleA.jl")
    elseif var == "Evaluate in ModuleB."
        include("ModuleB.jl")
    end
end

end # module MyPackage

./MyPackage/src/ModuleA.jl
module A
    using ..MyPackage: Model, getdata, isentrypoint, models
    processdataA(model::Model) = println(getdata(model))

    function __init__()
        if isentrypoint()
            println(getdata(models["Evaluate in ModuleA."]))
            println("We are evaluating in ModuleA.")
        end
    end
end


./MyPackage/src/ModuleB.jl
module B
   using MyModel: Model, getdata
   using ..MyPackage: models, isentrypoint
   processdataB(model::Model) = println(getdata(model))

   function __init__()
        if isentrypoint()
            processdataB(models["Evaluate in ModuleB."])
            println("We are evaluating in ModuleB.")
        end
    end
end

./MyPackage/Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.3"
manifest_format = "2.0"
project_hash = "a583b77fe794703b105dd8a19da10f8b2723c0b5"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.MyModel]]
path = "../MyModel"
uuid = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
version = "0.1.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

./MyPackage/config.toml
runtimevars=["Evaluate in ModuleA.", "Evaluate in ModuleB."]

./MyModel
cat: ./MyModel: Is a directory
./MyModel/Project.toml
name = "MyModel"
uuid = "99c7e603-0d09-40a3-8af7-b79e4bce3e38"
authors = ["root "]
version = "0.1.0"

./MyModel/src
cat: ./MyModel/src: Is a directory
./MyModel/src/MyModel.jl
module MyModel

struct Model
data::String
end

getdata(model::Model) = model.data

end # module MyModel

Executing…

$ julia --project=MyPackage MyPackage/src/MyPackage.jl
Evaluate in ModuleA.
We are evaluating in ModuleA.
Evaluate in ModuleB.
We are evaluating in ModuleB.

Loading as a library:

$ cd MyPackage && julia --project -e "using MyPackage"
# no output

Having formatting issues. Will fix later. Scroll right

Interactive usage:

julia> using Pkg

julia> pkg"activate MyPackage"                                  Activating project at `~/hijit/MyPackage`
                                                              
julia> using MyPackage

julia> model = MyPackage.models["Evaluate in ModuleA."]
MyModel.Model("Evaluate in ModuleA.")

julia> MyPackage.A.processdataA(model)
Evaluate in ModuleA.

[hijit]

Thanks so much for taking the time to write out the solution, yes I was a bit sloppy with the example in the beginning, I forgot to make the module declaration in both files. This answer is above and beyond.

I’m going to go ahead and mark this as the solution, because that is what it clearly is, but @nsajko made a number of comments that were helpful to my understanding, and clearly my initial formulation of my question was less then developed, and not nearly as clear and succinct as it could of been. So I’ll take the time to recap my question and summarize the important points of clarification from @nsajko.

My main question boiled down to the following:

How do I create three modules with a common local dependency whereby all objects and methods of one module can be handled by another module, without making the presence of any of these modules necessary to the use/import of any of the others?

To which @nsajko responded:

@nsajko

So you want to be able to use the modules independently from each other? Then you need to organize them into multiple packages.

Which is exactly what is done here. Anyways props for using the __ init __() methods of the module to obtain the appropriate behavior based on whether the program is executed from the entry point or not, very cool.

[mkitti]

Statements made at the top level would only be executed once during the “precompilation” stage of a package. The result would then be cached. Printing, however, would only occur once, during precompilation.

Statements the __init__() function will run everytime the module loads. That is either when you include it and or when you do using MyPackage.

Note that I established the relationship between MyPackage and MyModel by using Pkg.develop which records the location in the Manifest.toml of MyPackage. An alternative is to modify the LOAD_PATH.

Modules A and B are submodules of the package module MyPackage. In A, I imported Model and getdata from the parent package. In B, I imported the symbols from the MyModel package directly.

Note that LOAD_PATH modification allows for a simpler over file structure but is not recommended.

[nsajko]

@hijit, just learned about some things regarding the module situation in Julia, I think you might be interested:

1. relevant issue: relative using/import should search current directory · Issue #4600 · JuliaLang/julia · GitHub

2. one proposed solution is available as a registered package, FromFile: https://juliahub.com/ui/Packages/General/FromFile

So try out FromFile, maybe you find it helpful.