# Introductive material on Bayesian Networks inference / probabilistic programming?

**URL:** <https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017>\
**Category:** Probabilistic Programming\
**Created:** [October 16, 2023, 2:09pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017 "2023-10-16T14:09:55Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [October 16, 2023, 2:09pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/1 "2023-10-16T14:09:55Z")

</div>

Hello, any (gentle) introductory material to suggest on the topic of Bayesian Networks and probabilistic programming ? Or knowledge I need in advance in order to understand it ?

Context: I have clear the concepts of distribution, random variable, joint/marginal/conditional, Bayesian theorem and also prior/posterior distributions and likelihood, but I am still at odd to understand even the objectives of probabilistic programming or how Bayesian networks can be estimated from data and how predictions can be generated.  
The Turing.jl example and documentation is already too specific for me (the prologue “If you are already well-versed in probabilistic programming” unfortunately doesn’t apply to my case) and books like [“An Introduction to Probabilistic Programming”](https://arxiv.org/abs/1809.10756) seems both too wide and too theoretical (I can’t even understand the notation on the first few pages)…  
On the wikipedia page on [Bayesian Networks](https://en.wikipedia.org/wiki/Bayesian_network) I can follow until they start dealing with interventional questions, than I am lost (by the way, no reference to Turing.jl is given in that page)

Are probabilistic programming languages “tools” to express and fit a Bayesian Network ?

So, you can really see I need a very “gentle” introduction on the topic (conditional to my “level” 🙂 🙂 ) !!

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 16, 2023, 2:28pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/2 "2023-10-16T14:28:55Z")

</div>

One of the nicest introductions in my opinion, with practical examples:

> **[Bayesian Methods for Hackers](https://dataorigami.net/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)**
>
> Bayesian Methods for Hackers : An intro to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view.

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [October 16, 2023, 2:43pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/3 "2023-10-16T14:43:52Z")

</div>

Thanks … nice and really from the basic… a pity it has been ported to TensorFlow but not on Turing… ok, I guess that after it it will be relatively quick to move knowledge from PyMC to Turing 🙂

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [October 16, 2023, 8:04pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/4 "2023-10-16T20:04:17Z")

</div>

There is a (partial) port of [Bayesian methods for Hackers](https://github.com/abid8042/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers-in-Julia) to Julia.  
A good introduction to Bayesian networks is the chapter on _graphical models_ in the [ML book by Bishop](https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/). While some earlier attempts on probabilistic programming had been based on graphical models – exploiting their structure for inference by message-passing – modern sampling methods – such as NUTS implemented in Turing or PyMC – are more general and only require a differentiable function computing the log joint probability \log p(\mathrm{data}, \mathrm{parameters}).

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 16, 2023, 10:11pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/5 "2023-10-16T22:11:50Z")

</div>

[Statistical Rethinking](https://www.amazon.com/Statistical-Rethinking-Bayesian-Examples-Chapman-dp-036713991X/dp/036713991X/) is well-regarded for Bayesian inference.

---

<div class="post-metadata">

**Author:** ![Kangaa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kangaa/32/51765_2.png) [@Kangaa](https://discourse.julialang.org/u/Kangaa)\
**Post date:** [October 17, 2023, 6:02am UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/6 "2023-10-17T06:02:23Z")

</div>

In addition to seconding Statistical Rethinking, I recently found [this](https://gaussianbp.github.io/) great explainer of belief propagation on Bayesian networks.  
[https://gaussianbp.github.io/](https://gaussianbp.github.io/)

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [October 17, 2023, 8:29am UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/7 "2023-10-17T08:29:11Z")

</div>

Ok, I have now two main options to start with, “[Bayesian methods for Hackers](https://github.com/abid8042/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers-in-Julia)” and "[Statistical Rethinking](https://www.amazon.com/Statistical-Rethinking-Bayesian-Examples-Chapman-dp-036713991X/dp/036713991X/)".

Who knows them both, how do they compare, including in scope ?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 17, 2023, 8:36am UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/8 "2023-10-17T08:36:29Z")

</div>

I know both and wouldn’t recommend statistical rethinking as an intro book.

---

<div class="post-metadata">

**Author:** ![hackandthink](https://avatars.discourse-cdn.com/v4/letter/h/b782af/32.png) [@hackandthink](https://discourse.julialang.org/u/hackandthink)\
**Post date:** [October 17, 2023, 10:26am UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/9 "2023-10-17T10:26:41Z")

</div>

I liked this: Probabilistic Models of Cognition

[https://probmods.org/](https://probmods.org/)

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [October 17, 2023, 11:52am UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/10 "2023-10-17T11:52:52Z")

</div>

I know them both and my recommendation is always Statistical Rethinking. It’s really easy to read and gives you a good overview of Bayesian modelling and how to express them in a probabilistic programming language. I also enjoy the humorous tone in the book. 😊

---

<div class="post-metadata">

**Author:** ![math4mad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/math4mad/32/49486_2.png) [@math4mad](https://discourse.julialang.org/u/math4mad)\
**Post date:** [October 20, 2023, 11:10pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/11 "2023-10-20T23:10:50Z")

</div>

There are lots of source about Bayesian Stat  
here is a brief list, not all

## Book

[How to become a Bayesian in eight easy steps: An annotated reading list](https://link.springer.com/article/10.3758/s13423-017-1317-5)

the author has a good excellent blog about basic concept of bayes stat ,here it is : [https://alexanderetz.com](https://alexanderetz.com/2015/04/15/understanding-bayes-a-look-at-the-likelihood/)

[Bayesian Reasoning and Machine Learning | Higher Education from Cambridge](https://www.cambridge.org/highereducation/books/bayesian-reasoning-and-machine-learning/37DAFA214EEE41064543384033D2ECF0)

[Bayesian Cognitive Modeling](https://www.cambridge.org/core/books/bayesian-cognitive-modeling/B477C799F1DB4EBB06F4EBAFBFD2C28B)

## Turing.jl

If you want to learn about `Turing.jl` package, you can reference

- [`How to use Turing`](https://storopoli.io/Bayesian-Julia/pages/04_Turing/)
- [`hakank-bayes-code`](http://www.hakank.org/julia/turing/)
- [`mc-stan`](https://mc-stan.org/users/documentation/), cause turing’s syntax like stan, actually there are package `stan.jl`, and `mc-stan` has a good community.

---

<div class="post-metadata">

**Author:** ![rdiaz02](https://avatars.discourse-cdn.com/v4/letter/r/d78d45/32.png) [@rdiaz02](https://discourse.julialang.org/u/rdiaz02)\
**Post date:** [October 22, 2023, 5:26pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/12 "2023-10-22T17:26:26Z")

</div>

Another partial list 🙂

I’d start with Frank Wood’s presentation [https://www.cs.ubc.ca/~schmidtm/Courses/540-W18/wood.pdf](https://www.cs.ubc.ca/~schmidtm/Courses/540-W18/wood.pdf) , which gives a great overview of Probabilistic Programming (no need to go over all the 82 pages, and note that the language for the examples is Anglican). Frank Wood is one of the authors of the book “An Introduction to Probabilistic Programming” you mentioned. And I’d then read the first chapter of that book, skipping whatever seems over your head on a first reading.

Much easier than “An introduction to probabilistic programming”, and with examples you get to play with right away, is the fantastic “Probabilistic Models of Cognition”, [https://probmods.org/](https://probmods.org/), mentioned by @hackandthink. Håkan Kjellerstrand has written many of those models in Turing in [My Julia Turing.jl (probabilistic programming) page](http://www.hakank.org/julia/turing/), another great resource already mentioned by @math4mad. (Note, though, that at the time Håkan wrote his code, I think some facilities now available in Turing did not yet exist, or were as well documented, so a few things might be doable in a slightly more idiomatic “turingesque” way as of October 2023).

A comparison of some Bayesian statistics books, including “Statistical Rethinking” (as well as “A Student’s Guide to Bayesian Statistics”, “Bayesian Data Analysis”, and “Regression and Other Stories”) is available here

> **[Book Review of 5 Applied Bayesian Statistics Books — LessWrong](https://www.lesswrong.com/posts/5EqGPCKvm6m9XC7JJ/book-review-of-5-applied-bayesian-statistics-books)**
>
> There are 5 strong contender for best Bayesian Statistics book …

(not everybody agrees with everything said there, but I think it is a balanced overview. Oh, and if you decide to give BDA a go, Aki Vehtari has a course based on it: [Bayesian Data Analysis course](https://avehtari.github.io/BDA_course_Aalto/) )

Anyway, I’d definitely start with Frank Wood’s presentation and the first chapter of their book: it will answer some of your questions (“the objectives of probabilistic programming”) and help you anticipate and understand the differences of focus and emphasis between, say, “Statistical Rethinking” and “Regression and other stories” on the one hand and “Probabilistic models of cognition” (or “An Introduction to Probabilistic Programming”) on the other.

---

<div class="post-metadata">

**Author:** ![pleiby](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pleiby/32/8355_2.png) [@pleiby](https://discourse.julialang.org/u/pleiby)\
**Post date:** [October 22, 2023, 6:01pm UTC](https://discourse.julialang.org/t/introductive-material-on-bayesian-networks-inference-probabilistic-programming/105017/13 "2023-10-22T18:01:27Z")

</div>

[Tutorial Series 09 | Julia Probabilistic Programming for Beginners](https://www.youtube.com/playlist?list=PLhQ2JMBcfAsgU7kZ-Ee_SDrjhJIehICmR) is a wonderful, fairly recent entry this year by “doggo dot jl”. A video series with accompanying repository of notebooks, this module follows some of McElreath’s excellent, Statistical Rethinking book/lectures, but is an even more succinct and approachable (gentle) introduction, and is implemented in Julia/Turing.jl. Really enjoyable. Then you can follow this with some of the other resources mentioned above. (Actually, I strongly recommend anything by [doggo dot.jl](https://www.youtube.com/@doggodotjl). Created with the wide-eyed, basic perspective of an inquisitive amateur, they efficiently explore many powerful aspects of the “vast julia wilderness,” in a fun, thoughtful manner.) Good luck!