Turn a module into a package locally

This topic has been asked many times. I have read many posts. Much of the information is incomplete or contradictory. I have read the package manual. It uses the word “project” in 2 very different senses. So, let’s not use that word–it is more of an informal notion. The manual also focuses on packages that are in a git repo. I use git locally for version management. I keep a sort of backup on github, but I am not systematic about dev vs. master branches. I realize that to publish the package in the Julia package directory I’d have to clear that up with discipline.

I just want to be able to do : using mything and have this work with Revise while I am still developing and testing the work.

It is a module in a Julia source file in a directory with other source files and some essential external files for loading essential parameters. The module and the julia source file have the same name (though the file has the .jl extension of course). It seems this is necessary; others say it is not. Confusing.

The module has using for 8 packages. The module has includes for 6 source files. The 6 source files are, naturally in the same directory with the source file containing the module definition.

This works: push!(LOAD_PATH, "/Users/lewis/.julia-local-packages/Covid") in my startup.jl file. Note that the directory is called Covid and the module is called CovidSim and the source file is called CovidSim.jl. It is symlinked from the directory where I work on it (this is what I and most IDEs would call a project–a place where all the stuff is, with or without extra metadata describing/defining the stuff).

I was told I could just do:

] dev <directory containing the source file for the module>

This, of course, doesn’t work. It is missing steps like generate and activate. I see that I could do add, which might work. But, Revise says that I mustn’t do that because Revise will not see changes to the package in that case. And I want Revise to work. The problem is that I have a module and some source files. I do not have a package.

First, then is to create a package with a Manifest.TOML file, a source directory, all in some special place. While I am still developing the “package” it would be inconvenient if I need to copy to that special place. If it is special (like in a specific place with a specific name like “source” that’s easy: I’ll just symlink it.

It is fine if the answer is just stick to pushing to LOAD_PATH. That has zero overhead and does everything I want.

When it is time to really make a package (if ever), then I’ll build it by hand carefully following the manual. I understand and accept that much of the package overhead is there for good reasons: enabling widespread distribution of packages RELIABLY and securely; providing dependency management; and providing users with super simple access to packages and management of packages (add, rm, up, etc.). All of this works really well; has enabled bootstrapping a large Julia ecosystem quickly; and has avoided lots of headaches that other language ecosystems have had. I just need a little workflow clarification.

1 Like

Please don’t post in so many threads about this. Since you have decided to make a new thread, I am reposting my reply here:

With regards to the definition of project, I agree with other posters, a project is just an environment.

It sounds like a frustrating experience. Let me give you my workflow to see if it helps you. As I understand it you two pieces of code

  1. A Package, which you want to re-use across many projects
  2. A Project, here meaning a set of code which has it’s own environment but is not meant to be re-used.

At the end of the day, I think you should have a folder structure like this.

Documents
 ├── Packages
 |      └── MyPackage
 |              ├── src
 |              └── Project.toml
 ├── Projects
 |       └── MyProject
 |              ├── src
 |              └── Project.toml

To do that, go into your Packages folder. This can be in Documents or wherever you want. It does not need to be in your .julia folder. Do ] generate MyPackage .

Next close out of julia. Go to your Projects folder and do the same command . You still want to ] generate . So you do ] generate MyProject .

Now you want to let your MyProject know that it can use code from MyPackage . To do this, go into your MyProject folder and do ] activate . . This will create a new environment. I think this might be where you are running into trouble.

Now your package repl should look like

(MyProject) pkg>

What you want do next is

] dev '~/Documents/Packages/MyPackage`

Now using MyPackage should work and you can work on both in tandem.

1 Like

Additionally, It is not clear what you want to edit using Revise, exactly. Would you like to edit MyProject or edit MyPackage?

1 Like

It’s definitely necessary, and I don’t know who would have told you otherwise. If you can find a link, then we can correct whatever misinformation is out there.

This is easy to verify:

  • Create an empty folder and cd into it.
  • Create a single Foo.jl file containing module Foo:
$ echo "module Foo; end" >> Foo.jl
  • Start Julia add the current directory to the LOAD_PATH and then load the module:
julia> push!(LOAD_PATH, @__DIR__)
4-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/user/folder"

julia> using Foo
[ Info: Precompiling Foo [top-level]
  • Now rename Foo.jl to NotFoo.jl, so the module name does not match the file:
$ mv Foo.jl NotFoo.jl
  • And try again:
julia> push!(LOAD_PATH, @__DIR__)
4-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/user/folder"

julia> using NotFoo
[ Info: Precompiling NotFoo [top-level]
ERROR: KeyError: key NotFoo [top-level] not found

Perhaps someone at some point told you that you can have nested modules whose names are not the same as the top-level module? That’s true–you can have Foo.jl with:

module Foo

  module InnerModule
  ...
  end
end
3 Likes

Thanks.

It was a discourse comment. I think the point is missing altogether in documentation. I’ll re-read and post an edit if it is really missing.

The source code…

Thanks for all the suggestions.

Maybe my question wasn’t clear and we should just forget about “project”. It seems to mean the environment for one’s Julia installation. Or maybe the environment notion that Pkg permits so that dependencies can be collected in once place
just for one bit of work. Like a virtualenv in Python. It’s not as clear as it might be, but it’s not a problem. And, I can see that it is needed when working on different “projects” (in the informal sense) with different dependencies.

I want to make a local package and keep working on the code IN THE PACKAGE. One day, it’s really a bit self-contained and seems to fit the description of “application”. Too many words for too many concepts—let’s not go there. It’s just
about 3000 lines of code and needs to be split across source files and a module in a package
seems like the easiest way to treat that as one coherent whole. A module would be fine, but there are issues with including a module; a package seems cleaner.

You just need to do the first bit of what @pdeffebach said then (or what I said in the other thread). Do that, it will generate some files/folders. Inside the /src folder that gets generated, that’s where you put your code. It can take pretty much any form except you do need to have a .jl file that has the same name as the package, which defines the module itself. Suppose you’ve stuffed your code into this folder - now your package is created.

You still haven’t told the Julia package manager anything about this package (even though you used Julia to create it). So you need to

] dev '~/Documents/Packages/MyPackage`

like @pdeffebach said. This tells Julia that you are currently working on revising some part of this package.

Now when you want to actually do some development (with Revise because why not do it the easy way)

using Revise
using MyPackage
# test your functionality, make changes, etc
2 Likes

That’s really clear. Thanks.

A note on my above post. I don’t think I realized that you want to keep all the code in one folder tree. This should be very easy to do with Revise. I have a folder MyProject that meets the qualifications of a Package, i.e. it has a Project.toml and inside src there is a file called MyProject.jl. This means that using MyProject works when I start julia with julia --project (Or call Pkg.activate(".")). My folder structure looks liks:

shell> tree
.
├── Project.toml
└── src
    ├── Mod2.jl
    └── MyProject.jl

1 directory, 3 files

MyProject.jl looks like this:

module MyProject

export Mod2

greet() = print("Hello World!")

include("Mod2.jl")

end # module

Notice that nowhere do we write using Mod2. However we can access it from REPL after using MyProject.

Importantly, Revise works with this folder structure. If we do edit("src/MyProject.jl") and make changes to Mod2.jl these changes will show up.

If you think that Mod2 doesn’t contain functionality that is needed outside of MyProject and you don’t think Mod2 needs it’s own dependency management, this is a perfectly good workflow.

I think what was tripping you up was that you expected using Mod2 to work. using looks for things in LOAD_PATH and your Project.toml etc. Neither of those things know about Mod2, which is fine. And nor should they since it’s just a name-space used in MyProject.

If you have many Mod2.jl-type files scattered throughout your system, there is nothing stopping you from includeing them. But as other people have mentioned this is a recipe for lots of bugs and dependency hell.

In conclusion

  1. Everything in 1 repo called MyName? Make that repo a Package, meaning it has the src/MyName.jl file. You can organize your code through many modules in the code.
  2. Multiple modules you want to use each without a full package structure? As a stopgap, you can include them in src/MyName.jl but it’s better to bite the bullet and make all those Mod.jl files packages in their own right.
1 Like