Hello, any (gentle) introductory material to suggest on the topic of Bayesian Networks and probabilistic programming ? Or knowledge I need in advance in order to understand it ?
Context: I have clear the concepts of distribution, random variable, joint/marginal/conditional, Bayesian theorem and also prior/posterior distributions and likelihood, but I am still at odd to understand even the objectives of probabilistic programming or how Bayesian networks can be estimated from data and how predictions can be generated.
The Turing.jl example and documentation is already too specific for me (the prologue “If you are already well-versed in probabilistic programming” unfortunately doesn’t apply to my case) and books like “An Introduction to Probabilistic Programming” seems both too wide and too theoretical (I can’t even understand the notation on the first few pages)…
On the wikipedia page on Bayesian Networks I can follow until they start dealing with interventional questions, than I am lost (by the way, no reference to Turing.jl is given in that page)
Are probabilistic programming languages “tools” to express and fit a Bayesian Network ?
So, you can really see I need a very “gentle” introduction on the topic (conditional to my “level” ) !!
Thanks … nice and really from the basic… a pity it has been ported to TensorFlow but not on Turing… ok, I guess that after it it will be relatively quick to move knowledge from PyMC to Turing
There is a (partial) port of Bayesian methods for Hackers to Julia.
A good introduction to Bayesian networks is the chapter on graphical models in the ML book by Bishop. While some earlier attempts on probabilistic programming had been based on graphical models – exploiting their structure for inference by message-passing – modern sampling methods – such as NUTS implemented in Turing or PyMC – are more general and only require a differentiable function computing the log joint probability \log p(\mathrm{data}, \mathrm{parameters}).
In addition to seconding Statistical Rethinking, I recently found this great explainer of belief propagation on Bayesian networks. https://gaussianbp.github.io/
I know them both and my recommendation is always Statistical Rethinking. It’s really easy to read and gives you a good overview of Bayesian modelling and how to express them in a probabilistic programming language. I also enjoy the humorous tone in the book.
I’d start with Frank Wood’s presentation https://www.cs.ubc.ca/~schmidtm/Courses/540-W18/wood.pdf , which gives a great overview of Probabilistic Programming (no need to go over all the 82 pages, and note that the language for the examples is Anglican). Frank Wood is one of the authors of the book “An Introduction to Probabilistic Programming” you mentioned. And I’d then read the first chapter of that book, skipping whatever seems over your head on a first reading.
Much easier than “An introduction to probabilistic programming”, and with examples you get to play with right away, is the fantastic “Probabilistic Models of Cognition”, https://probmods.org/, mentioned by @hackandthink. Håkan Kjellerstrand has written many of those models in Turing in My Julia Turing.jl (probabilistic programming) page, another great resource already mentioned by @math4mad. (Note, though, that at the time Håkan wrote his code, I think some facilities now available in Turing did not yet exist, or were as well documented, so a few things might be doable in a slightly more idiomatic “turingesque” way as of October 2023).
A comparison of some Bayesian statistics books, including “Statistical Rethinking” (as well as “A Student’s Guide to Bayesian Statistics”, “Bayesian Data Analysis”, and “Regression and Other Stories”) is available here
(not everybody agrees with everything said there, but I think it is a balanced overview. Oh, and if you decide to give BDA a go, Aki Vehtari has a course based on it: Bayesian Data Analysis course )
Anyway, I’d definitely start with Frank Wood’s presentation and the first chapter of their book: it will answer some of your questions (“the objectives of probabilistic programming”) and help you anticipate and understand the differences of focus and emphasis between, say, “Statistical Rethinking” and “Regression and other stories” on the one hand and “Probabilistic models of cognition” (or “An Introduction to Probabilistic Programming”) on the other.
Tutorial Series 09 | Julia Probabilistic Programming for Beginners is a wonderful, fairly recent entry this year by “doggo dot jl”. A video series with accompanying repository of notebooks, this module follows some of McElreath’s excellent, Statistical Rethinking book/lectures, but is an even more succinct and approachable (gentle) introduction, and is implemented in Julia/Turing.jl. Really enjoyable. Then you can follow this with some of the other resources mentioned above. (Actually, I strongly recommend anything by doggo dot.jl. Created with the wide-eyed, basic perspective of an inquisitive amateur, they efficiently explore many powerful aspects of the “vast julia wilderness,” in a fun, thoughtful manner.) Good luck!
