The introduction of TermInterface.jl has allowed for large potential in generalization of term rewriting and symbolic analysis and manipulation features. Also, it’s been a few months we’ve been talking about the integration between Metatheory.jl (e-graph rewriting) with Symbolics.jl, and it has been shown in the Symbolics.jl paper.
I’m now working in my thesis to finalize this integration and show some results, but there’s a few practical software engineering things that should be considered before continuing this integration. I made some diagrams trying to make a recap of the current situation of the symbolics ecosystem now, and how it might evolve in the near future. Many PRs integrating those packages are still open.
I’d love to hear your comments and ideas for future development.
Even if CI suites seem well-defined, splitting into smaller chunks always runs the risk of downstream breakage. The way to run a smooth ecosystem is to assume you’re never perfect, and make a system that battles your human imperfections. So I would be weary of splitting too small, but one way to combat that is to ensure that we do extensive downstream testing and enforce that if you cause downstream breakage you follow up with downstream PRs.
I do think there needs to be symbolic commons, but I wonder if having the rewriters and the term interface and the patterns etc. really all needs separate repos, or whether that is cohesively the common interface package itself. For every split, the question should be why. Why could these two packages be put together. Do they introduce a mess of dependency growth to something downstream that only needs one vs the other? Does it introduce a large amount of using time? If those questions are no, and they sit in the same general realm, the default should be to bring them together, not to split. That will decrease maintenance, using time, increase precompilation ease, and make things just overall easier.
I totally agree with you on this point chris. What is your opinion about the last ‘layered’ scenario where we move functionality down to MT?
It seems the most reasonable solution to me, and I’d be eager to start working on this ASAP to make this a good chapter of my thesis. When writing the chapter introducing the packages i also noticed that an excessive splitting of the modules complicated the structure of the chapter, like “what package should i introduce first?”, instead, if we use this layered approach I could linearly explain the process from high level to low level, and then going back to how we used egraphs to optimize the Symbolics.jl codegen.
Also, this would imply additional PRs and breakage in other dependant packages, and I’m not sure if @shashi is happy with this, but this is better than excessive fragmentation, too many CI configs and downstream breakages. In this layered approach tho, the overall exposed APIs can be maintained as similar to the ones that we’re lowering from SymbolicUtils.jl to Metatheory.jl (e.g. no name changes), such that PRs to downstream such as ModelingToolkit could only add an “using Metatheory, Symbolics” and most probably work.
I think that such an integration needs general consensus of the people using and developing the said packages. Also, I think that it won’t take much time (a few days at max) if we plan the CI structure correctly and work together on this task.
Remember the bottom of the stack needs to be the most careful with dependencies. I think that could work though, the current list isn’t too bad. If you wish to move Metatheory.jl to JuliaSymbolics (having the bottom of the stack not under community management would be questionable for longevity).
I’m generally ok with the last approach proposed here after talking to Alessandro at length about it. Provided:
Let’s try to have it so that it is easier to move Rewriters/Rules/Patterns out of Metatheory if need be – this means no abstraction leaks even within Metatheory. There’s one spot in the Rule type where we’d need to some compromises as we discussed, and which is why a clean split into a separate package is not possible. Let’s try to not have other compromises!
I don’t want the file structure to be fragmented more like in your PR, would be great if you could leave file structure as they were in SU, and as much as possible the files themselves unchanged… This is just a matter of preference for me and Yingbo to be able to maintain the code. We use vim and like big files we can search with /.
I hope to have no breaking changes in SU from this!
