Local Modules with Conflicting Names and Module Management

General Usage
1

Hello everyone, I am writing a midsize project that has quite a few modules and I just thought of a potential problem that I cannot see answered anywhere.

Suppose I want to add two distinct sources to my LOAD_PATH for modules “/path1/” and “/path2/”.
Suppose both path1 and path2 contain separate and distinct modules with the name “CustomModule.jl”.

Calling using CustomModule will result in a name conflict. I assume that Julia will use the first module it comes across that matches the criteria, but this allows for the possibility for large projects to have severe structural issues when trying to manage modules. Is there a way to manage LOAD_PATH that will avoid this trouble? I thought of requiring modules to ALWAYS be submodules, but this is quite a hackey solution.

Alternatively, is there a way to do something like using "relativePath/customModule"? I want to avoid including the same module in different files because the types will technically be distinct and hence will lead to errors.

Thanks

Edit:

I found that even my method of trying to use my modules from the LOAD_PATH is having another issue: primarily the fact that my module’s guaranteed return types are being changed to Main.Type instead of just Type. I could fix this by simply making no guarantees on return types, but this seems like a poor solution.

2

This is not a problem that I’ve ever run into when using Julia, and I think there are good ways to avoid it. It’s hard to know exactly what your case looks like, but if path1 and path2 are both collections of Julia modules, then you could, for example, make them modules on their own. For example, if you create:

module Path1

  module CustomModule.jl
    f() = "path1"
  end

end

module Path2
  module CustomModule.jl
    f() = "path2"
  end
end

then there will be no ambiguity. You can always unambiguously refer to Path1.CustomModule or Path2.CustomModule.

Note that this does not mean you need all of those modules in one file. You can organize the files themselves however you see fit, using include() to include the relevant pieces.

Can you elaborate on this a little bit? I’m not sure what the problem is, but it sounds like there may be something unusual in the way you are loading your code.

3

Thank you for the above response, I will try and give you a more clear idea of what my intention has been.

Here is the simple file that I have decided to create as a manager for my files so that each module using another module actually refer to the exact same Type. Each of the folders in moduleContainingFolderList contains Julia modules. This module is run before anyothers to correctly add to LOAD_PATH.

#This module will modify LOAD_PATH to include all module directories
root_path = @__DIR__
push!(LOAD_PATH, root_path)

moduleContainingFolderList = [
    "/gamemanagement",
    "/heuristics",
    "/representation",
    "/util"
]

for i = 1:length(moduleContainingFolderList)
    path = root_path * moduleContainingFolderList[i]
    if !(path in LOAD_PATH)
        push!(LOAD_PATH, path)
    end
end

I am then able to use using moduleName freely, but the problem was that all of the functions and types loaded this way seem to be entering the Main module, so that when I attempt to run my code I get “cannot convert” errors. Before I run my code, the module getting access to by using is getting pre-compiled. After some further testing here, it seems like it was not ‘recompiling’ my “CallTimer.jl” after it was modified, but this still does not explain to me why it was ever being loaded as part of Main? Originally I was going to call it Timer, but I got a name conflict with Base, so perhaps this caused the error? The only way I can seem to get changes to be reflected across modules once they have been compiled is to completely restart Julia in ATOM, and run my LOAD_PATH modifying file again.

As you will see, i tried adding include_dependency("./../util/CallTimer.jl") to get it to check file versions, but no luck.

Below are my actual modules, the CallTimer is in the folder “Amazons/util”, but the GameManager is in “Amazons/gamemanagement”.

module CallTimer

    export TargetTimer, Flag, startTimer, getNewFlagTimer

    struct TargetTimer
        timeToElapse::Float64
        target
        f #f is the function to call on target after elapsed time
    end

    #This method should be called with the @async macro to run in the back
    function startTimer(timer::TargetTimer)
        sleep(timer.timeToElapse)
        timer.f(timer.target)
    end

    mutable struct Flag
        isRaised::Bool
    end

    function raiseFlag(flag::Flag)
        flag.isRaised = true
    end

    function getNewFlagTimer(time::Float64)::Tuple{TargetTimer, Flag}
        flag = Flag(false)
        timer = TargetTimer(time, flag, raiseFlag)
        println(typeof((timer, flag)))
        return (timer, flag)
    end

end

module GameManager
    include_dependency("./../util/CallTimer.jl")
    using Board
    using CallTimer

    mutable struct Game
        masterBoard::BoardState
        currentPlayerTurn::Piece
        maxTimePerMove::Int
    end

    function slowCount(flagObj::Flag)
        i = 0
        while !(flagObj.flag) && i < 100
            sleep(1)
            i+=1
            println(i)
        end
    end

    (timer, flag) = getNewFlagTimer(5.0)
    @async startTimer(timer)
    slowCount(flag)

end

I know this is quite a horrendously long post, so thank you very much for your time.

Edit: Fixed a variable name, was relatively unimportant.

4

There are a few things in your example that indicate that you’re doing things in an unusual way:

  1. Dynamically modifying LOAD_PATH in a complicated way is almost never necessary (it has never been necessary in my experience).
  2. Using include_dependency is also quite unusual, and should not be necessary when including Julia source code under normal circumstances (I have never needed it and did not even know it existed until your post).

Instead, I would suggest making your folder structure look like this:

.
├── GameManagement
│   └── src
│       ├── GameManagement.jl
│       └── Internal.jl
└── Heuristics
    └── src
        ├── Heuristics.jl
        └── Internal.jl

where the files look like:

# file: GameManagement/src/GameManagement.jl 
module GameManagement

using Heuristics

# "Internal" is a submodule, defined in `Internal.jl`. The name
# "Internal" here is just an example; you should name your modules
# however you see fit. 
include("Internal.jl")
using .Internal


function f()
  # Use g() from the local Internal.jl
  Internal.g()
  # and also call something from Heuristics
  Heuristics.f()
  
  # you can even call something from Heuristics.Internal if you need 
  Heuristics.Internal.g()
end

end

# file GameManagement/src/Internal.jl 
module Internal  # the name "Internal" is just an example

g() = println("game management internal")

end

# file Heuristics/src/Heuristics.jl 
module Heuristics

# "Internal" is a submodule, defined in `Internal.jl`. The name
# "Internal" here is just an example; you should name your modules
# however you see fit. 
include("Internal.jl")
using .Internal

# You can reference functions inside the Internal submodule unambiguously
f() = Internal.g()

end

# file Heuristics/src/Internal.jl 
module Internal # the name "Internal" is just an example

g() = println("heuristics internal")

end

If you set up your code in this way, there is no need to do anything with LOAD_PATH except to add the single folder containing GameManagement, Heuristics, Utils, etc.

Julia will be able to find Heuristics within GameManagement with no trouble and confusion about which types are which:

julia> push!(LOAD_PATH, @__DIR__)
4-element Array{String,1}:
 "@"                                 
 "@v#.#"                             
 "@stdlib"                           
 "/home/rdeits/Downloads/test/"

julia> using GameManagement

julia> GameManagement.f()
game management internal
heuristics internal
heuristics internal

julia> using Heuristics

# Verify that the module `Heuristics` that was loaded inside `GameManagement` is
# exactly the same one we got from `using Heuristics` at the REPL:
julia> GameManagement.Heuristics === Heuristics
true

Not only does this avoid name clashes and accidental duplicate module definitions, it also makes it easier to eventually transition your code to a full-fledged package so that others can install and use it without needing to do anything to LOAD_PATH at all.

2 Likes
5

Okay, that seems way nicer than what I have been doing, but I cannot seem to get it to work as you have it laid out. I get the following error when I try and run the line using Representation inside of GameManager.jl.

ERROR: LoadError: ArgumentError: Package Representation not found in current path:
- Run `import Pkg; Pkg.add("Representation")` to install the Representation package.

I have altered my file structure to be like this and included the modules you have defined above.

.
├── GameManagement
│   └── src
│       ├── GameManager.jl
│       ├── GameManagent.jl
│       └── Internal.jl
├──  Heuristics
│    └── src
│        ├── History.jl
│        ├── GameManagent.jl
│        ├── Internal.jl
│        ├── Mobility.jl
│        ├── Territory.jl
│        └── Transposition.jl
├──  Representation
│    └── src
│        ├── Board.jl
└── Util
│    └── src
│        └── CallTimer.jl

I have been avoiding using “include” because I receive errors about the types defined in modules not being equivalent to those outside of it. I am working on Linux (Ubuntu); could this be a Linux only bug?

I also added the modules and code that you have written exactly as specified, but get the following output.

julia> push!(LOAD_PATH, @__DIR__)
4-element Array{String,1}:
 "@"                                      
 "@v#.#"                                  
 "@stdlib"                                
 "/home/daniel/Programming/Julia/Amazons/"

julia> using GameManagement
[ Info: Precompiling GameManagement [top-level]
WARNING: --output requested, but no modules defined during run
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for GameManagement [top-level]
│   exception = ArgumentError: Invalid header in cache file /home/daniel/.juliapro/JuliaPro_v1.1.1.1/compiled/v1.1/GameManagement.ji.
└ @ Base loading.jl:969
ERROR: KeyError: key GameManagement [top-level] not found
Stacktrace:
 [1] getindex at ./dict.jl:478 [inlined]
 [2] root_module at ./loading.jl:898 [inlined]
 [3] require(::Base.PkgId) at ./loading.jl:864
 [4] require(::Module, ::Symbol) at ./loading.jl:853

I did navigate to that directory and delete the file of GameManagement.ji that was there in hopes that it was simply corrupted, but no such luck.

I appreciate all the help you have given me, but do not feel obligated to continue.

6

using Foo will look in your load path for Foo.jl or Foo/src/Foo.jl, and expects to find a module called Foo in that file. Your Board.jl should probably just be named Representation.jl.

The last error you posted is odd. Perhaps it’s the same issue? Your GameManagement.jl should begin:

module GameManagement

....
2 Likes
7

Thank you so much for all the help, I did not know that about the required module structure.

Here is how I resolved/suspect the cause of my two other errors. A module that had previously been explicitly written with include_dependency() had since been re-named and moved. I am unsure of where or why this was causing an error, so I ended up deleting the entire folder DEPOT_PATH[1]/compiled''' with rm -rf compiled``` on my command line. This alone did not resolve the issue. I am uncertain as to why, but changing:

module Timer
    export TargetTimer, Flag, startTimer, getNewFlagTimer
   ...
end

to (by adding a space)

module Timer

    export TargetTimer, Flag, startTimer, getNewFlagTimer
   ...
end

seems to have resolved my issue. I have tried to reproduce the issue by removing the space and restarting julia and atom, but to no avail. It also came to my attention during this process that if module A is used in a module B (and is precompiled), then the only way that I have gotten these changes to be reflected is by restarting Julia.

8

The space is definitely not necessary, but it is entirely possible that you got Julia into a weird state and touching that file caused a recompilation that fixed the underlying issue.

Check out Revise.jl for a much easier way to hot-reload code without restarting Julia.

2 Likes
9

Alright, will do. Thanks gain.