These ideas are in the direction of dealing with some of the issues encountered when developing large modular packages. Currently (I agree) that the flexibility of the module system of Julia has some trade-offs relative to more rigid models like Python’s.
Disclaimer: I have already mentioned that these ideas may not even be worth taken seriously. I am just winding up my imagination and I have no idea which would be the implementation difficulties and consequences. Yet, neither of the ideas seem to be breaking changes, at first sight.
Packages that contain other packages, all of them available from the general registry.
The idea would be able to have one package, lets say MyBigPackage registered in the general registry. But this package would be able to contain (parent?) other packages (MySmallPackage1, MySmallPackage2…), which would also have individual uuids. The big package would be installed with ] add MyBigPackage, installing all small packages as dependencies, but each small package could also be installed independently with: ] add MyBigPackage.MySmallPackage1 and used with using MyBigPackage.MySmallPackage1 even if MyBigPackage is not installed.
Why: a) One would be able to develop MyBigPackage with a modularity that would have all the advantages of the dependency system and modularity of regular dependencies. No more “includes” if one does not want to. b) The small packages would be available. c) This could alleviate the “name shortage” of packages. I would be able to have my MyBigPackage.GramSchmidt package which would be potentially be available as a dependency to any other package (particularly other of my packages), without reserving the GramSchmidt name.
Perhaps the MyBigPackageProject.toml file could contain the list of the child packages.
Modules that behave like Pluto notebooks (no conflicting names, no code order)
I see some potential advantages in terms of debbuging and code organization if I could define a module that would be Pluto like in those senses. Meaning:
@pluto_like module A
f(x::T) = 1
struct T
x
end
end
no code order (this is bad style, but still useful many times). But most importantly:
@pluto_like module A
f(x::Int) = 1
f(x::Int) = 2
end
This would throw an error (multiple definitions, as in a Pluto notebook). Since Pluto already interprets the code like that, maybe that macro is easier than I thought.
That is not to say that such module definitions would be a bad idea, I’m just adding what I have run into a few times with Pluto. In Pluto all the code is close by, and I know about this issue, so I find it easy to fix and to work with this in mind when using/writing my own macros
The first idea is basically package namespacing, isn’t it? I also think such lightweight namespacing could often be useful. Potential examples among existing packages:
RecipesBase → Plots.RecipesBase so that it’s clear what the recipes are for
Optim → Optim (almost dependency-free), Optim.Newton (additionally depends on e.g. PositiveFactorizations), …
I think there was a discussion of something similar to this proposal and one of the obstacles was modules/packages possible conflicts. For example, RecipeBase, should it’s functions be defined in module RecipeBase or they should be defined in the module Plot.RecipeBase?
In the first case, you can have a potential conflict between Plot.RecipeBase and MyOwnPlotNamespace.RecipeBase. In the second case RecipeBase of Plot should somehow know about the existence of Plot module and be able to hack its way into it, which can lead to another set of problems.
Well, I definitely didn’t think much about specific details of how this should work on the technical level.
Maybe another separator for namespaced packages instead of . (dot) can solve the issues you refer to?
Can we register the three packages there independently in the general registry? Can two different packages point to different directories of the same GitHub repository? And, of course, have the these packages dependent on each other (I don’t mean circularly)
(The only thing this simplifies is not having to have multiple GitHub repositories for something that is effectively one big package).
They are relatively new (the issue is this one I believe), but a simple Pkg.add("MyPackageUtils") would work in all Julia versions (1.0+) once it is registered; in Julia 1.5, Pkg.jl has the tools to do advanced things like add them at the master branch. You register them with
for example. The Julia GitHub action for tests supports them (you pass project setting to point to the Project.toml of the package you want to test), as does the processcoverage action (which has a directories parameter for where to look for coverage files).
Yep, I think this is answered by the above.
Yeah, they are treated the same as packages from different repos, so dependencies are fine. For development, if you wanted you could check in a Manifest to dev say MyPackageUtils in MyPackage by relative path so you can jointly develop them but that does have the downsides of checking in a Manifest; for example, you need to keep it updated and CI will use it, so it can be hard to run CI on multiple Julia verisons. But there are options like multiple manifests, see e.g. https://github.com/JuliaRegistries/General/tree/master/.ci, and tooling can make this easier.
This is great. I my humble opinion this addresses many of the issues some people have with modules, with the additional benefit of making the modules truly useful as standalone units.
Yeah, I agree! That’s what I was trying to say in the other thread but I think I didn’t do so very clearly. I think with that + LocalRegistry.jl (both in fact thanks to lots of work by @GunnarFarneback) one can easily use packages as independent components for open source work in General and for private development or development in a lab registry like the HolyLabRegistry.