DifferentialEquations.jl/JuliaDiffEq is a prime example here since it started with a Plots.jl like optional dependency structure and switched to a modular library structure. There are ups and downs, but at this point I very much agree that developing everything with smaller packages and glue packages + putting it all together in a large metapackage is the way to go. Not only are there no precompilation problems, but then you can also separate your tests and lower CI loads by doing so. And you can more easily “isolate a problem”: version bounding a specific bad library update. Lastly, contributors seem to find it much easier to dive into little bites rather than the whole library.
That said, test breakage can be an issue. You can end up with chicken-and-egg situations. For example, say you developed a feature in a specific package, but it has grown to be something useful elsewhere. You may want to move an abstract type from a leaf package to a core/base package. If both packages are not on master, tests will fail in this situation. Since our package setup tests with dependencies on release, this ends up with some situations where tests are guaranteed to fail until tags go through or you locally check masters together. Also, changes to a Base/core package can require changes in the leafs. Adding a new type parameter can cause many leafs to need an upgrade.
A more common problem for me is that I like to have a lot of integration tests. The Base package has test dependencies on some of the leaf packages because they are used to make sure everything works still. But this leads to a chicken and egg test problem with larger Base changes (though it’s just a test problem, and not actually failing). I could see Plots having this kind of issue, since you’d probably want to test PyPlotPlots with Plots.jl to make sure a bunch of features work, but now changing Plots can require a PyPlotPlots update for tests to pass.
The issue here is that these things lead to upper bounds when doing updates, and Pkg cannot handle those well. That’s the root of this issue:
That said, these are mostly tooling problems. And actually many (most?) software ecosystems probably wouldn’t have these kinds of problems. For example, Bio.jl is already very modular in its own package, and so it can split and probably not have many tests overlap. But if Pkg and the CI setups was made to more easily handle this kind of changes, I think that this is the clear preferred path. Putting all of the packages together in an org also makes it easy to share privileges and manage them as a group.
So after commenting for a long time asking for condition modules, I don’t really think they are as necessary since Julia does so well with glue packages. I just think the testing and tagging structure needs to be updated to accommodate working like this in more complicated setups.