What is Bayesian A/B Testing and how does it compare to Frequentist hypothesis testing?

Bayesian A/B testing models treatment effects as probability distributions, updating beliefs about which variant is superior as data arrives, directly answering "What is the probability that B is better than A?" In contrast, Frequentist hypothesis testing evaluates the probability of observing data as extreme as, or more extreme than, what was measured, assuming a null hypothesis is true, typically yielding a p-value.

Here’s a comparison of their core approaches:

| Feature | Bayesian A/B Testing | Frequentist A/B Testing |
| :---------------- | :---------------------------------------------------- | :------------------------------------------------------- |
| Philosophy | Updates prior beliefs with data to form posterior beliefs (P(Hypothesis | Data)). | Assesses data extremity under a null hypothesis (P(Data | Null Hypothesis)). |
| Output | Posterior probability distribution for each variant; direct probability of one variant being better (e.g., P(Conversion_B > Conversion_A) = 0.95). | p-value, confidence intervals; reject or fail to reject a null hypothesis (e.g., p < 0.05). |
| Stopping | Flexible; stop when credible interval for difference excludes zero or P(B > A) reaches a desired threshold (e.g., 99%). | Requires pre-defined sample size or sequential testing adjustments to maintain Type I error rates. |
| Prior Info | Explicitly incorporates prior knowledge or historical data into the model. | Does not explicitly incorporate prior information; relies solely on current experiment data. |
| Interpretation| Intuitive: "There is a 95% chance variant B is better than A." | Counter-intuitive: "If the null hypothesis were true, there's a 5% chance of observing data this extreme or more extreme." |

For implementation, Bayesian A/B tests often use libraries like PyMC or Pyro for probabilistic programming. A typical workflow involves defining a model with priors for conversion rates (e.g., pm.Beta distribution), observing data, and then sampling from the posterior distribution to derive insights. For instance, to compare two conversion rates:

import pymc as pm
import numpy as np

# Simulate data
n_a, conv_a = 1000, 150 # Variant A: 1000 visitors, 150 conversions
n_b, conv_b = 1000, 180 # Variant B: 1000 visitors, 180 conversions

with pm.Model() as model:
    # Priors for conversion rates
    # Using uninformative Beta(1,1) prior (uniform between 0 and 1)
    theta_a = pm.Beta("theta_a", 1, 1)
    theta_b = pm.Beta("theta_b", 1, 1)
    
    # Likelihood (Binomial for conversions)
    obs_a = pm.Binomial("obs_a", n=n_a, p=theta_a, observed=conv_a)
    obs_b = pm.Binomial("obs_b", n=n_b, p=theta_b, observed=conv_b)
    
    # Define the difference in conversion rates
    diff_conversion = pm.Deterministic("diff_conversion", theta_b - theta_a)
    
    # Sample from the posterior
    trace = pm.sample(2000, tune=1000, random_seed=42, return_inferencedata=True)

# Calculate the probability that B is better than A
prob_b_better_a = (trace.posterior["diff_conversion"] > 0).mean().item()
print(f"Probability that B is better than A: {prob_b_better_a:.3f}")

Frequentist tests, conversely, often leverage scipy.stats for proportion tests (e.g., scipy.stats.proportions_ztest). You would define a significance level (alpha, commonly 0.05) and compare the resulting p-value to this threshold to make a decision.