According to Registrator.jl#287 and Pkg.jl#1422 merged pull requests, several packages can now be organized as subdirectories of a single repo. However, I have not been able to found any detailed docs, which would explain how to use this new functionality. As a result, I would like to ask a number of questions here, on discourse.
How are the subdirectories supposed to be organized? Must each of the subdirectories have the normal package structure with Project.toml, Manifest.toml, ./src, ./test, ./docs etc.?
Is it possible to have shared docs or tests for the packages in subdirectories?
If the packages in subdirectories depend on each other, how imports should be organized?
Does subdirectory need to be registered so that other packages can use it?
How does one register a subdirectory package? How does one add a subdirectory package to an active environment?
Let’s say I want to split some core functionality of a package into a subdirectory: I have SomePackage and I want to split it into subdirectories SomePackageBase and SomePackage. Is it possible to reregister SomePackage so that it points to a subdirectory now?
How does one deal with the versions of the subdirectory packages? As far as I understand, github allows one to tag the whole repository. Does it mean that all the subdirectory packages need to have the same versions? How does one alter TagBot setup?
It seems to create chicken and egg problem in the scenario where you separate core functionality of SomePackage into SomePackageBase: Since SomePackageBase contains core functionalty, SomePackage would depend on it. However, it can not import it until SomePackageBase is registered.
It seems to me that, until SomePackageBase is registered, you would have your master branch in a broken state, where you can’t even run the tests. Is such a problem really inevitable?
Git tags have no consequence for Julia versions. The version number is decided by Project.toml and the registry addresses versions by content hashes (specifically git tree hashes), not git tag names. Relevant git tags are helpful for humans interacting with the repository, but that’s all.
Do as you would if SomePackageBase lived in a separate repository, make it work and register it, then switch SomePackage over to use it.
I must admit that being able to house multiple packages in these new “multi-package bundle repository” is a great organizational tool that does greatly ease the number of commits & pushes a developer needs to execute (Thanks to everyone who worked on this).
(And @tim.holy: Please correct me if I am mis-interpreting your point here.)
if you want to bundle up & deploy multiple packages with a long string of dependencies, you still need to manually sequence up the registration process of all these packages.
For what it’s worth, I do think this is a likely scenario because personally, I would tend to use a “multi-package bundle repository” to “bundle” together a cluster of related packages with potentially tight dependencies.
Thanks for pointing that out. I was unaware that building local registries like this was a practical means of registering multiple packages simultaneously.
At some point, I thought I saw either @GunnarFarneback or @fredrikekre suggest that an automated flow could eventually be developed to achieve simultaneous registration. I wonder if the methodology you just outlined is the goal, or merely a temporary stepping stone (looking at a roughly 1 year timeframe, I suppose).
I noticed the SnoopCompile.jl example you referenced only had to register a chain of 2 packages: SnoopCompileCore & SnoopCompile.
Would you say that this methodology is more convenient than registering the 2 packages manually? I ask because it almost seems as if the local registry solution only becomes the more convenient approach once we need to register a chain of 3 or 4+ packages. It might also depend on how long it takes for github CI to greenlight your builds (if you tend to be extra diligent).
I wonder if the methodology you just outlined is the goal, or merely a temporary stepping stone
The latter, I think
(looking at a roughly 1 year timeframe, I suppose).
It could probably be pulled off in a couple of days if someone decided to invest the time. It doesn’t require changes to Julia or Pkg, I think, I imagine it is only a matter of changes to JuliaRegistrator.
Would you say that this methodology is more convenient than registering the 2 packages manually?
For two packages, no, and lately I have been registering them sequentially. I think I started it before JuliaRegistrator could register subdirectory packages.