Can a module in a package itself be a package?

[sadish-d]

I have a pacakge (PackageCore) that provides the main functionality of another (PackageFull). I’ve decided that I want to be able to develop and test these as independent packages. This is the structure I am currently using:

PackageFull
	src
		PackageFull.jl
	test
		runtests.jl
	Project.toml
	Manifest.toml
	PackageCore
		src
			PackageCore.jl
		test
			runtests.jl
		Project.toml
		Manifest.toml

When I develop PacakgeFull, I do ]dev "path/to/PackageFull". When I develop PacakgeCore, I do ]dev "path/top/PackageFull/PackageCore".

The important bit is: I don’t add/dev PackageCore as a package/dependency from PackageFull. Rather, in PackageFull.jl I just include("../PackageCore/PackageCore.jl"). In other words, for PackageFull, PackageCore is just a module defined within module PackageFull ... end, not a package.

Is there anything wrong with this setup?

[kellertuer]

The on fear I see, that with the include instead of a dependency, you basically hide the dependency. Because in practice, you actually do depend on having all functions fro your core package.

That is a bit of cheating in my view. but independent of that, you might confuse a user, that looks at the dependencies and thinks “ah, it does not depend on the core package” – and then wonders that it indeed does (just in a very hidden way, because to see the dependency he/she has to actually find the exact code line that does the include.

So, do you have a good reason for that include instead of a dependency?

In my setup (Manifolds.jl and ManifoldsBase.jl) we have separate repositories (that is not necessary, you can have them in the same I think), but I usually just have both packages in development mode.

[sadish-d]

I think it hides dependency in the sense that no all the source code used in PackageFull is contained within its src folder, which is the norm others might be used to. But any dependency on external packages would show in both the Project.toml files. In any case, this is not meant to be an external package, so let’s say for now that I am not too concerned about breaking norm.

I don’t have a good reason for why I am thinking of structuring my project this way. It’s more of a why not. In my head PackageFull encapsulates PackageCore. I understand that it “feels” off, but I don’t see how this structure has more problems than having two seperate packages self-contained in their own folders. But then again, I have little experience with software development and version control.

[kimikage]

If you really want to adopt the include strategy, it would be better to name the folder packagecore. I think just that much confusion will be greatly reduced.

I would not recommend that strategy, though.

[kimikage]

”In principle", there is no guarantee that the PackageCore exists there, since the source files in the PackageCore/ is not known from the PackageFull/Project.toml.
Future versions of julia or Pkg.jl may offer an option to not check out folders such as test or docs to reduce the weight.

[nsajko]

Can a module in a package itself be a package?

No, but a single repo may have multiple packages in subdirectories. This is supported and used in some registered packages.

@sadish-d

I don’t add/dev PackageCore as a package/dependency from PackageFull. Rather, in PackageFull.jl I just include("../PackageCore/PackageCore.jl").

Sure, that works, but why not just have two packages? That way their relationship is explicit, so they’re more reusable.

[sadish-d]

I had a very specific question in mind, but I came across a few things that I’m going to just leave here in case someone else finds them useful.

Turns out there are decent people doing all sorts of wild things when it comes to nested packages. Looks like ModelParameters.jl does the inverse of what I mentioned, equivalent to putting PackageFull is in a subfolder of PackageCore: GitHub - rafaqz/ModelParameters.jl: Easy, standardised parameter get/set for heterogeneous or nested immutable models.

Something I hadn’t thought of is how the folder structure is going to interact with repositories. One approach suggests having a Repo folder, where the packages are in their own subfolder, but then the repo has a Project.toml and Manifest.toml: Feature request: scan subdirs for updated versions/projects · Issue #1874 · JuliaLang/Pkg.jl · GitHub

And more discussion here:

Usage of subdirectories to store multiple packages in a single repo

Thanks for pointing that out. I was unaware that building local registries like this was a practical means of registering multiple packages simultaneously. At some point, I thought I saw either @GunnarFarneback or @fredrikekre suggest that an automated flow could eventually be developed to achieve simultaneous registration. I wonder if the methodology you just outlined is the goal, or merely a temporary stepping stone (looking at a roughly 1 year timeframe, I suppose). I noticed the SnoopComp…

I don’t know where I’ll land. Some paths are more well trodden than others and the approach of using include() does not seem to be one of them.

[nsajko]

@sadish-d

Some paths are more well trodden than others and the approach of using include() does not seem to be one of them.

It’s basically a terminology issue. Including files is fine, just don’t call an include-d file a package. A package has a Project.toml file (almost never should Manifest.toml be commited to Git, BTW), and if you depend on it you just put it into your Project.toml and then import it with using or import.