Magic incantations to make a local module work

I have some complex code that makes more sense as a module. I have done this before with 2 other “local”–as in in my file system–modules and everything works fine. This one just won’t work.

The filename and the module name match. I have noted that this is a requirement that is not documented.

# filename CovidSim.jl

module CovidSim

<code>

end # module CovidSim

This all exists in a directory called “Covid”. I have symlinked this directory into the directory: /users/<myname>/.julia-local-packages

My startup.jl file looks like:

push!(LOAD_PATH, "/Users/<myname>/.julia-local-packages/Covid")
push!(LOAD_PATH, "/Users/<myname>/.julia-local-packages/nn by hand")
push!(LOAD_PATH, "/Users/<myname>/.julia-local-packages/TOML")

try
    using OhMyREPL
catch
    @warn "OhMyREPL not installed"
end

When I examine the LOAD_PATH environment variable, it is:

julia> LOAD_PATH
6-element Array{String,1}:
 "@"                                            
 "@v#.#"                                        
 "@stdlib"                                      
 "/Users/<myname>/.julia-local-packages/Covid"     
 "/Users/<myname>/.julia-local-packages/nn by hand"
 "/Users/<myname>/.julia-local-packages/TOML"      

When I do using CovidSim there is a bit of a delay and the prompt returns. But, nothing I have exported can be accessed. I cannot access functions qualifying the names such as CovidSim.run_a_sim

It’s as though nothing has happened. Flummoxed.

This housekeeping stuff just gets exhausting and takes lots of time.

Solved.

Simple syntax error.

One export statement had a trailing comma after the items

A simple syntax error can cause problems anywhere. Error messages and side effects don’t always make it obvious.

Didn’t take long to find.

This is still a little bit of a fidgety process though all works fine and the fault was all mine.

Instead of explicitly manipulating LOAD_PATH, it would be easier to directly rely upon Pkg’s features.
In this case, you should probably “develop” your package:

julia> #hit "]" to enter pkg REPL mode
(@v1.4) pkg> dev /Users/<myname>/.julia-local-packages/Covid

From there on, you won’t have to manually fiddle with LOAD_PATH: when you’re in your default environment, using Covid will just work.

4 Likes

That’s awesome. Not portable across machines, but trivial. I assume there is no reason that the project directory containing the module needs to be in any magic directory.

Looks like it requires Julia 1.4 so time to bite that bullet.

Indeed, you can dev any path to a local directory containing the package, and things will work seamlessly.

And this has worked for a long time; there is no need to use Julia 1.4 for this.

1 Like

By the way, it is actually not required that the file name matches the module name. In fact, you can have multiple modules in one file. In Python, every file is also a module. Julia intentionally avoids this restriction.

1 Like

…hmmm. Then, how does Julia find the module within a project directory where the directory name is arbitrary? Does Julia open every .jl file and look for a module statement—tricky, because there can be many, as you point out?

The package and module name spaces are a little confusing.

As far as I know, you can only use the package manager to manage packages, not arbitrary source files with modules in them. So there are two approaches you could take:

  1. Manually source the files in your REPL session (or in your startup.jl file) with include("myfile.jl").
  2. Turn your modules into small packages and manage them with the package manager.

It’s pretty easy to wrap your module into a package. Refer to the docs for more info on this.

Package namespaces are actually just modules. Here is the source code for Example.jl:

module Example
export hello, domath

"""
    hello(who::String)
Return "Hello, `who`".
"""
hello(who::String) = "Hello, $who"

"""
    domath(x::Number)
Return `x + 5`.
"""
domath(x::Number) = x + 5

end

So you can see that the namespace for the Example.jl package is just created with a module. :slight_smile:

2 Likes

But, it’s not a package. This does not work. It is a file containing a module, which does a bunch of using’s and includes. generate does not seem to work because it doesn’t ask for enough arguments. It should request a name and a directory for the files.

There is LOTS of conflicting advice and direction around this. One way and one way that works is how this stuff should be. Usually there are not enough benefits to the alternatives to justify having so many ways.

It sounds like a frustrating experience. Let me give you my workflow to see if it helps you. As I understand it you two pieces of code

  1. A Package, which you want to re-use across many projects
  2. A Project, here meaning a set of code which has it’s own environment but is not meant to be re-used.

At the end of the day, I think you should have a folder structure like this.

Documents
 ├── Packages
 |      └── MyPackage
 |              ├── src
 |              └── Project.toml
 ├── Projects
 |       └── MyProject
 |              ├── src
 |              └── Project.toml

To do that, go into your Packages folder. This can be in Documents or wherever you want. It does not need to be in your .julia folder. Do ] generate MyPackage.

Next close out of julia. Go to your Projects folder and do the same command. You still want to ] generate. So you do ] generate MyProject.

Now you want to let your MyProject know that it can use code from MyPackage. To do this, go into your MyProject folder and do ] activate .. This will create a new environment. I think this might be where you are running into trouble.

Now your package repl should look like

(MyProject) pkg>

What you want do next is

] dev '~/Documents/Packages/MyPackage`

Now using MyPackage should work and you can work on both in tandem.

2 Likes

It is enough to make it a project.

You can always rename/move the directory.

There is a recommended way: make a project, and activate it, and work in that context. Which is the advice various people have given above. It is not conflicting.

I don’t know where you got the idea to use LOAD_PATH from, but I am guessing you must have seen it in a reply from years ago on some discussion forums. It used to be a workaround, but it is not the recommended way since the new Pkg(3) was introduced.

2 Likes