Creating multi-module projects (preparing for Pkg3)

General Usage
, , ,
1

I’m having a hard time understanding how to “properly” lay out semi-shared code among a series of mini-projects, especially in the context of preparing for Julia v0.7/Pkg3. (I’m just starting the project, so with v0.7 imminent, I don’t want to lock myself in to the v0.6 way of doing things, even though I can’t actually use v0.7 consistently yet due to packages I need being not yet compatible.)

The basic idea is that I have one top-level directory which encompasses a whole project — the exploration of various research topics — for just me personally. There’s potential for some amount of code reuse — which I imagine are submodules of this directory — as well as serial-numbered/chronologically-ordered one-off scripts which drive through to some result. My imagined layout ends up looking something like:

Research.jl
├── PlotFuncs
│   └── src
│       └── PlotFuncs.jl
├── src
│   ├── Research.jl
│   └── paths.jl
├── ToyPixPixCov
│   └── src
│       ├── fft_tools.jl
│       ├── matrixpipe.jl
│       ├── ToyPixPixCov.jl
│       └── toysims.jl
├── S0001_proj_algorithms.ipynb
├── S0002_purif_params.ipynb
└── S0002_purif_params.jl

where there exist three modules — the top-level (generic) Research and two sub-projects ToyPixPixCov and PlotFuncs — and the one-off scripts are each of the SXXXX-prefixed *.jl or *.ipynb files.

Now to the problem. Say I try setting this up using Julia v0.7 + Pkg3, and include in one of the sub-modules a dependency on one of the other modules in this directory:

pkg> generate Research
┌ Info: Pkg3 is running without precompile statements, first action will be slow.
│ Rebuild julia with the environment variable `JULIA_PKG3_PRECOMPILE` set to enable precompilation of Pkg3.
└ This message can be disabled by setting the env variable `JULIA_PKG3_DISABLE_PRECOMPILE_WARNING`.
Generating project Research:
    Research/Project.toml
    Research/src/Research.jl

shell> cd Research
/tmp/Research

pkg> generate ToyPixPixCov
Generating project ToyPixPixCov:
    ToyPixPixCov/Project.toml
    ToyPixPixCov/src/ToyPixPixCov.jl

pkg> develop ToyPixPixCov
 Resolving package versions...
  Updating `Project.toml`
 [9c9cea7e] + ToyPixPixCov v0.1.0 [`ToyPixPixCov`]
  Updating `Manifest.toml`
 [9c9cea7e] + ToyPixPixCov v0.1.0 [`ToyPixPixCov`]

pkg> generate PlotFuncs
Generating project PlotFuncs:
    PlotFuncs/Project.toml
    PlotFuncs/src/PlotFuncs.jl

shell> sed -i -e 's/^greet().*$/using ToyPixPixCov\nToyPixPixCov.greet()/g' PlotFuncs/src/PlotFuncs.jl

pkg> develop PlotFuncs
 Resolving package versions...
  Updating `Project.toml`
 [b2d8554e] + PlotFuncs v0.1.0 [`PlotFuncs`]
  Updating `Manifest.toml`
 [b2d8554e] + PlotFuncs v0.1.0 [`PlotFuncs`]

Now trying to load the two modules, I can load ToyPixPixCov since it relies on no other modules, but I get an error with PlotFuncs saying it can’t find ToyPixPixCov:

julia> using ToyPixPixCov

julia> using PlotFuncs
ERROR: LoadError: ArgumentError: Module ToyPixPixCov not found in current path.
Run `Pkg.add("ToyPixPixCov")` to install the ToyPixPixCov package.
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:869
 [2] include at ./boot.jl:306 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1072
 [4] _require(::Base.PkgId) at ./loading.jl:998
 [5] require(::Base.PkgId) at ./loading.jl:879
 [6] require(::Module, ::Symbol) at ./loading.jl:874
in expression starting at /tmp/Research/PlotFuncs/src/PlotFuncs.jl:3

If I switch to Julia v0.6, I can get around most of this by just adding the current working directory to LOAD_PATH:

julia> VERSION
v"0.6.2"

julia> push!(LOAD_PATH, pwd())
3-element Array{Any,1}:
 "/usr/local/share/julia/site/v0.6"
 "/usr/share/julia/site/v0.6"      
 "/tmp/Research"                   

julia> using ToyPixPixCov

julia> using PlotFuncs
Hello World!
julia>

(Of course, to load Resarch on v0.6 I also need to do push!(LOAD_PATH, realpath("..")) first, while that does just work on v0.7+Pkg3…)

I realize things are still very much in flux and a work in progress, but any guidance on how to accomplish this?

2

In Pkg3, a package can only load modules that it declares a dependency on. The problem right now is that PlotFuncs does not declare a dependency on ToyPixPixCov.

Looking in the Manifest.toml file

[[PlotFuncs]]
path = "PlotFuncs"
uuid = "76fbcee0-2b78-11e8-1ecf-498917db9923"
version = "0.1.0"

[[ToyPixPixCov]]
path = "ToyPixPixCov"
uuid = "59c18c18-2b78-11e8-0607-81faa5decbcf"
version = "0.1.0"

We have the two packages but there is no deps entry for ToyPixPixCov in the PlotFuncs table.

If we manually change this file to:

[[PlotFuncs]]
deps = ["ToyPixPixCov"]
path = "PlotFuncs"
uuid = "76fbcee0-2b78-11e8-1ecf-498917db9923"
version = "0.1.0"

[[ToyPixPixCov]]
path = "ToyPixPixCov"
uuid = "59c18c18-2b78-11e8-0607-81faa5decbcf"
version = "0.1.0"

then everything works:

julia> using PlotFuncs
Hello World!

So we just need to add a way for Pkg3 package to specify dependencies (and we are actively working on that).

2 Likes
3

Ah, thanks. I’ve been stuck in cycles of trying to add the dependency to PlotFuncs’s Project.toml and Manifest.toml files, but that hasn’t worked out, either. So in a setup like this, is the correct thing to do is to only use the top-level Manifest.toml file (i.e. the ones generated when creating the sub-modules are irrelevant)?

1 Like
4

Eventual Manifest.toml and Project.toml in your “subpackages” are not used at all when you are loading packages from the “toplevel”. The manifest of the toplevel package contains the whole dependency graph of all the dependencies.

So the issue here is simply that Pkg3 is not done yet and declaring dependencies for new packages is one of the features missing.

1 Like
5

Awesome — I think I managed to wipe away all the confusion and setup something that works.

Thanks for (1) the quick response, and (2) the great work. (I’m really looking forward to the release :smiley: .)

1 Like
6

Thank you so much both for trying this out and for reporting back your experience. I can’t tell you how much it helps with figuring out where the Pkg3 tooling needs more work.

This is a pretty key aspect of package development that we hadn’t really hit yet since we’ve mostly been focused on getting things working for existing packages and this is about creating a new package and adding a dependency on it elsewhere. My immediate reaction is that the best way to handle this may be to prompt the user when there’s an import X or using X statement that doesn’t “resolve” and ask them if it should be made to work, and if there are multiple possible meanings of the name X, which one to use, and then update various project and manifest files in dev packages in order to make the statement resolve, and then give a summary of what was changed where. Does that seem like a UX flow that would have been good here?

Oh, also in the future, please don’t spend too much time beating your head against the wall with these things – post a new package manager question and we’ll be more than happy to answer it!

1 Like
7

Just be careful about reliance on prompts and menus, because not everyone is running Julia through the REPL (e.g. you could be using IJulia, or Juno/Atom, or vscode).

8

Yes, I think as part of this process we need to make sure that REPL interactivity has a proper API and that it is generic so that the appropriate prompting can be done in different graphical environments as well. Fortunately, I don’t think there’s too many different modes of interaction that are necessary – prompting for a set of choices is a pretty universal UI construct that should be possible everywhere, and is probably easier to implement in most graphical environments than in the terminal even. Still, it will be a bit of a design/implementation process.

9

My misunderstanding of the situation had me thinking(/guessing) that the setup process was hierarchical in some way — you’d add dependencies in each “subpackage” individually a la pkg> add Compat, at the top level you’d “register” your subpackages (the unknown I thought I was missing), and then pkg> update in the top level would then recurse through all the components, building the complete manifest.

Anyway, the bit of information about constructing Manifest.toml dependency entries by hand was enough to get me moving along, so thank you again!

1 Like