I think the biggest elephant in the room is how to get Gen and Turing talking.
Take the below with grains of salt - again, I’m one person, outside of both groups. I have no idea about the internal dynamics of these projects. But here are some of my opinions.
I’ve done a good amount of thinking about this. Part of the reason I implemented Jaynes was to really try and understand Gen at a deep level. I also implemented interfaces to Turing’s packages including AdvancedHMC to see what’s going on with interop there. I don’t claim that these interfaces are “well done” - only that I sort of see how things can work together.
Despite this, it’s my belief that it is totally possible to get these two packages working together. I honestly think it would send a really strong message if University of Cambridge and MIT partnered up to manage the JuliaPPL organization and we had an incredibly solid set of base packages. (again, not up to me, but an interesting proposal)
Here’s where I think this is going(and a couple of recent posts/work which has driven this viewpoint). Feel free to disagree, and send your own take!
There’s an open issue to modularize Gen
Hong is fully in support for abstract interfaces (see above).
There are some interesting ideas for a common IR
I’ve been trying to work out how to optimize dynamic programs at the IR level
To me, there’s no inherent barrier which prevents Gen and Turing from working together. Sure, it’s going to involve some sacrifices (likely on both sides) - fundamentally, I think the issue is going to be one of affiliation and reputation and all the sticky issues which come with that. And I totally understand why these issues exist! It makes complete sense why Gen is a separate project, and why Turing is a separate project. But when it comes to shared base packages, does this have to be the case? Can a partnership between MIT and University of Cambridge work this out and present a unified PPL front? I’m not in a position to say obviously, but it’s cool to consider.
I see this as fertile ground to identify what exactly each side “needs” to support their research ideas.
Here are my opinions (again, please refute):
The generative function interface is perfect for the ‘AbstractPPL.jl’ package. As far as I can tell, the ontology is very useful. And it supports arbitrary dynamic programs, all the way down to specialized static languages. This I think is the real innovation of Gen (as has been recently firmly drilled into me by the Gen squad). It allows for the construction of their inference DSLs. And it’s super easy to extend it to new model types through inheritance. It took me like 200 lines to go from Jaynes with crappy standard inference to Jaynes with involution DSL kernels courtesy of the interface.
The advanced samplers and bijectors are the real contribution of Turing. I do not say this to spit in the face of all the excellent work which has gone into the language and associated framework. This is simply my opinion - because I favor the extendability of what the generative function interface offers. I am, however, not familiar with the optimizations which @mohamed82008 has mentioned previously - so it’s possible that there are really cool things that I’m missing. I do know, however, that AdvancedHMC is really a work of art. So is MCMCChains from a usability and user-facing perspective. I’ve found it incredibly useful to hook these up to my trace-based experiments. And there’s a whole interesting area of “how do you visualize chains of traces”.
On the bijectors, I would like to see the Turing bijectors work merge with Marco’s work on trace translators (which generalize bijectors to distributions over choice maps, correct me if wrong).
I also see a rich area of inquiry related to @phg discussion of probabilistic IR. If it’s any indication of my interest, Jaynes implements the GFI - but it’s implemented by transforming and executing SSA IR. With this in mind, I can easily see the “core” of the org shaped as:
-
AbstractPPL - which includes the GFI, possibly also includes interfaces which Turing team agrees are relevant to extending what they would like to perform research in. I’m sure the GFI is useful - but this would generally be something which everyone needs some input on.
-
ProbabilisticIR - which extends/forks IRTools IR. The construction plan of this package is TBD depending on Julia 1.6. We could of course start now by working off IRTools IR - but we might have to accept some changes as new compiler interfaces stabilize.
-
AdvancedHMC - partially re-written to utilize interfaces agreed for AbstractPPL.
-
MCMCChains - same as above.
-
Bijectors/involution DSL - this merge I’m less certain about. This seems like a big lift/big thought piece to determine. I don’t think it’s crucial for now.
I’d also like to slot Jaynes in as an intermediate representation optimizer. Right now, this is GFI specific and works on IRTools IR (e.g. I’m hoping to transform it into a sort of optimizing compiler for the GFI). But I’m absolutely open to slicing this apart and re-constructing it if it fits the community plan, to support more general interfaces. I can’t say for certain if my work is useful yet - until I experiment more.
Anyways, ultimately something like this requires buy in from more than just myself. And ultimately, the people who have to buy in are more important than I am for a proposal like this. There are also many reasons why this sort of thing may be out of the question. But for the community, I’d like to raise these points.
Edit: just to be clear, this organization above does not include front-end interfaces for specifying models. I think generally this can be pretty flexible - if we have an agreed upon IR (whatever it is). My point of view comes from how Jaynes implements the GFI now. The interfaces transform the IR. To understand how to construct a front-end, we’d have to determine how you can implement the AbstractPPL interfaces as transformations on the IR. I know how to do this my way now, but there may be (likely, certainly are) other ways!
@phg @trappmartin @mohamed82008 @Marco_Cusumano-Towne @alex-lew @cscherrer