Benchmarking of PPL

Specific Domains Probabilistic programming
1

Hi all, I’ve seen the perf benchmark or sampling for Turing vs Stan but I was wondering if we have benchmarks for Pyro, pymc3, tensorflow-probabiliy with one chain and with multiple chains (since tfp seems to benefit from vectorizing chains?).

Thanks for the feedback!

1 Like
2

I think extending the Turing vs Stan benchmarks here should be relatively straight-forward. Also, we have a GSoC project for this so it may be worked on soon.

2 Likes
3

We have some benchmarks of ODE solvers in PPLs as well:

https://benchmarks.sciml.ai/html/ParameterEstimation/DiffEqBayesFitzHughNagumo.html

https://benchmarks.sciml.ai/html/ParameterEstimation/DiffEqBayesLotkaVolterra.html

Note that there’s a third benchmark which needs an update but Stan currently fails to be able to estimate it well, so for now we’re leaving that one off (we’ll need to decrease the timespan or something to make it easier to estimate for Stan).

1 Like
4

We have applications of GSoC students that will conduct benchmarks comparing Turing with various existing PPLs including pyro and TF-probability.

3 Likes
5

Thank you all for the answers, I’m looking forward to it!