Usage of subdirectories to store multiple packages in a single repo

[Gregstrq]

According to Registrator.jl#287 and Pkg.jl#1422 merged pull requests, several packages can now be organized as subdirectories of a single repo. However, I have not been able to found any detailed docs, which would explain how to use this new functionality. As a result, I would like to ask a number of questions here, on discourse.

• How are the subdirectories supposed to be organized? Must each of the subdirectories have the normal package structure with Project.toml, Manifest.toml, ./src, ./test, ./docs etc.?
• Is it possible to have shared docs or tests for the packages in subdirectories?
• If the packages in subdirectories depend on each other, how imports should be organized?
• Does subdirectory need to be registered so that other packages can use it?
• How does one register a subdirectory package? How does one add a subdirectory package to an active environment?
• Let’s say I want to split some core functionality of a package into a subdirectory: I have SomePackage and I want to split it into subdirectories SomePackageBase and SomePackage. Is it possible to reregister SomePackage so that it points to a subdirectory now?

The help is appreciated.

[fredrikekre]

Many/most of these questions can probably be answerd by this: Packages in subdirectories are just regular packages. They just happen to live in the same git repo.

---

@Gregstrq

How are the subdirectories supposed to be organized? Must each of the subdirectories have the normal package structure with Project.toml, Manifest.toml, ./src, ./test, ./docs etc.?

Yes they need to be packages on their own with Project.toml, src/Package.jl entrypoint etc. Probably this structure makes the most sense:

$ tree .
.
├── PackageA
│   ├── Project.toml
│   └── src
│       └── PackageA.jl
└── PackageB
    ├── Project.toml
    └── src
        └── PackageB.jl

4 directories, 4 files

or perhaps

$ tree .
.
└── packages
    ├── PackageA
    │   ├── Project.toml
    │   └── src
    │       └── PackageA.jl
    └── PackageB
        ├── Project.toml
        └── src
            └── PackageB.jl

5 directories, 4 files

@Gregstrq

Is it possible to have shared docs or tests for the packages in subdirectories?

Not sure what you mean – they are standalone packages. On the repo-level if you want some common tests you can structure it like this maybe:

$ tree .
.
├── docs
│   └── common_docs.html
├── packages
│   ├── PackageA
│   │   ├── Project.toml
│   │   └── src
│   │       └── PackageA.jl
│   └── PackageB
│       ├── Project.toml
│       └── src
│           └── PackageB.jl
└── test
    └── common_tests.jl

7 directories, 6 files

(but Pkg.jl doesn’t understand it so you must organize it yourself).

@Gregstrq

If the packages in subdirectories depend on each other, how imports should be organized?

Since they are standalone package the same mechanism as if they were different repos.

@Gregstrq

How does one register a subdirectory package?

Here is an example: InlineTest: bump version 0.1.0 -> 0.1.1 · JuliaTesting/ReTest.jl@57368e6 · GitHub

@Gregstrq

How does one add a subdirectory package to an active environment?

If it is registered just like a normal package. If you add by URL you specify the subdirectory after :: add url:/subdirpath

@Gregstrq

Let’s say I want to split some core functionality of a package into a subdirectory: I have SomePackage and I want to split it into subdirectories SomePackageBase and SomePackage. Is it possible to reregister SomePackage so that it points to a subdirectory now?

Yes.

[Raf]

@Gregstrq

it possible to reregister SomePackage so that it points to a subdirectory now?

It’s probably easier to put SomePackageBase in the subdirectory, you don’t have to change as many things - especially the repository name and URL.

Docs are the thing I would definitely combine, there is not real point having separate docs, it’s probably worse for users as well.

See ModelParameters.jl for an example of this setup - a subpackage included in the main docs:

GitHub - rafaqz/ModelParameters.jl: Easy, standardised parameter get/set for...

[Gregstrq]

@fredrikekre

@Gregstrq

Let’s say I want to split some core functionality of a package into a subdirectory: I have SomePackage and I want to split it into subdirectories SomePackageBase and SomePackage. Is it possible to reregister SomePackage so that it points to a subdirectory now?

Yes

Could you please point out what is the procedure to do that.

Also, are subdirectories supported by all Julia 1.x version, or is there some lower bound? Are subdirectories supported for the LTS version of Julia?

[GunnarFarneback]

@Gregstrq

Also, are subdirectories supported by all Julia 1.x version, or is there some lower bound? Are subdirectories supported for the LTS version of Julia?

All Julia 1.x versions can use registered packages that live in subdirectories but complete support (Pkg.dev and similar) requires Julia 1.5 or higher.

You can find some discussion related to this in Feature request: Store multiple registered packages in a single Git repository · Issue #1251 · JuliaLang/Pkg.jl · GitHub.

[Gregstrq]

@fredrikekre

Many/most of these questions can probably be answerd by this: Packages in subdirectories are just regular packages. They just happen to live in the same git repo.

How does one deal with the versions of the subdirectory packages? As far as I understand, github allows one to tag the whole repository. Does it mean that all the subdirectory packages need to have the same versions? How does one alter TagBot setup?

@fredrikekre

@Gregstrq

If the packages in subdirectories depend on each other, how imports should be organized?

Since they are standalone package the same mechanism as if they were different repos.

It seems to create chicken and egg problem in the scenario where you separate core functionality of SomePackage into SomePackageBase: Since SomePackageBase contains core functionalty, SomePackage would depend on it. However, it can not import it until SomePackageBase is registered.

It seems to me that, until SomePackageBase is registered, you would have your master branch in a broken state, where you can’t even run the tests. Is such a problem really inevitable?

[Henrique_Becker]

@Gregstrq

Since SomePackageBase contains core functionalty, SomePackage would depend on it. However, it can not import it until SomePackageBase is registered.

Maybe a hacky fix would include a Manifest for SomePackage temporarily, in which you point out that SomePackageBase is deved in a subdirectory.

[GunnarFarneback]

@Gregstrq

How does one deal with the versions of the subdirectory packages? As far as I understand, github allows one to tag the whole repository. Does it mean that all the subdirectory packages need to have the same versions? How does one alter TagBot setup?

Git tags have no consequence for Julia versions. The version number is decided by Project.toml and the registry addresses versions by content hashes (specifically git tree hashes), not git tag names. Relevant git tags are helpful for humans interacting with the repository, but that’s all.

@Gregstrq

It seems to create chicken and egg problem in the scenario where you separate core functionality of SomePackage into SomePackageBase: Since SomePackageBase contains core functionalty, SomePackage would depend on it. However, it can not import it until SomePackageBase is registered.

Do as you would if SomePackageBase lived in a separate repository, make it work and register it, then switch SomePackage over to use it.

[MA_Laforge]

@GunnarFarneback

Do as you would if SomePackageBase lived in a separate repository, make it work and register it, then switch SomePackage over to use it.

This reminds me of one of Tim’s more visual posts: Feature request: Store multiple registered packages in a single Git repository · Issue #1251 · JuliaLang/Pkg.jl · GitHub

(Image embedded to avoid unnecessary clicking)

I must admit that being able to house multiple packages in these new “multi-package bundle repository” is a great organizational tool that does greatly ease the number of commits & pushes a developer needs to execute (Thanks to everyone who worked on this).

But…
(And @tim.holy: Please correct me if I am mis-interpreting your point here.)

if you want to bundle up & deploy multiple packages with a long string of dependencies, you still need to manually sequence up the registration process of all these packages.

Unlikely scenario?

For what it’s worth, I do think this is a likely scenario because personally, I would tend to use a “multi-package bundle repository” to “bundle” together a cluster of related packages with potentially tight dependencies.

For example:

• Colors, ColorSchemes, ColorTypes, ColorBrewer, ColorVectorSpace, NamedColors

Even though, at this point in development, the interfaces for this particular “package bundle” are probably very stable, and a cascade of registrations would probably be infrequent.

Simultaneous registration?

My personal belief is that “package bundle” registration would be much simpler if all packages were registered simultaneously, using a common revision number.

• You do lose some flexibility in how independent packges in a bundle can be.
• It likely requires more complexity in Julia’s package manager implementation.

But:

• it map more directly onto Git tags (which point to a given repository state) — even if Julia’s package manager does not rely on Git tags.

And:

• if packages from a package bundle are not related enough for a single revision number to make sense — maybe those packages shouldn’t be in a single bundle.

[tim.holy]

You’re right that the registration step is still sequential if you use JuliaRegistrator. You can use @GunnarFarneback 's wonderful LocalRegistry:
https://github.com/GunnarFarneback/LocalRegistry.jl
to locally (on your machine) register all versions of all packages, and then submit a single PR to General manually. To do this, you’ll need a git checkout of the general registry.

An example of such a PR is here: https://github.com/JuliaRegistries/General/pull/23132. I squashed the changes made by LocalRegistries into a single commit before submitting it.

The main hangup: someone needs to manually merge the PR.

[MA_Laforge]

Thanks for pointing that out. I was unaware that building local registries like this was a practical means of registering multiple packages simultaneously.

At some point, I thought I saw either @GunnarFarneback or @fredrikekre suggest that an automated flow could eventually be developed to achieve simultaneous registration. I wonder if the methodology you just outlined is the goal, or merely a temporary stepping stone (looking at a roughly 1 year timeframe, I suppose).

@tim.holy

The main hangup: someone needs to manually merge the PR.

I noticed the SnoopCompile.jl example you referenced only had to register a chain of 2 packages: SnoopCompileCore & SnoopCompile.

Would you say that this methodology is more convenient than registering the 2 packages manually? I ask because it almost seems as if the local registry solution only becomes the more convenient approach once we need to register a chain of 3 or 4+ packages. It might also depend on how long it takes for github CI to greenlight your builds (if you tend to be extra diligent).

[tim.holy]

I wonder if the methodology you just outlined is the goal, or merely a temporary stepping stone

The latter, I think

(looking at a roughly 1 year timeframe, I suppose).

It could probably be pulled off in a couple of days if someone decided to invest the time. It doesn’t require changes to Julia or Pkg, I think, I imagine it is only a matter of changes to JuliaRegistrator.

Would you say that this methodology is more convenient than registering the 2 packages manually?

For two packages, no, and lately I have been registering them sequentially. I think I started it before JuliaRegistrator could register subdirectory packages.