[ANN] BayesianExperiments.jl: A Toolbox for Bayesian AB Testing Experiments

Package Announcements
,
1

I am excited to announce a new package: BayesianExperiments.jl. This package provides a toolbox for running various types of Bayesian AB testing experiments, including Conjugate prior models, Bayes factor models. Currently it supports stopping rules including expected loss, probability to beat all and Bayes factor.

The project starts from one of our internal projects by applying Bayesian AB testing. We find the methodology is attractive in many ways comparing to the Frequentist AB testing. But also there are some pain points that a well designed tool can help to solve.

If you are interested in AB testing/online experimentation with Bayesian statistics, please check out the repo, documentation, or play with the Jupyter notebook in Binder.

Any feedback, tips or comments are welcome!

14 Likes
2

I love it but a few comments from me.

  1. The examples in the documentation is not newbie friendly. By that I meant that if the reader is NOT familiar with Bayesian AB testing, they would be completely lost. Perhaps a very very simple example for those Bayesian newbies. Words like “ConjugateBernoulli” could be very scary for beginners. Perhaps you could use a uniform distribution for absolute beginners. Have at least one simple example for absolute beginners.
2 Likes
3

Thanks for the suggestion! I will definitely include an example to explain the basic concepts to the broader audience, assuming no prior knowledge on Bayesian AB testing. I am thinking it will be easier to explain the concepts if I could use more intuitive examples like the coin flipping example.

1 Like