A/B Test False Positives: p=0.03 with 50 Users Explained

The 200-Conversion Mirage

You ship a new checkout button. After 200 conversions, the p-value hits 0.03. Your manager celebrates. You push to production.

Two weeks later, the “winning” variant is underperforming the control by 8%. What happened?

The culprit isn’t bad luck — it’s a fundamental mismatch between frequentist hypothesis testing and small-sample reality. Most A/B test calculators assume you’re flipping a coin thousands of times. But early-stage products, B2B funnels, and niche features rarely see that volume. And that’s where the math breaks down in ways most data analysts never learned in school.

Why p < 0.05 Doesn’t Mean What You Think

The classical frequentist test answers: “If there were truly no difference, how often would I see a result this extreme?”

But that’s not the question you actually care about.

You want to know: “Given this data, what’s the probability variant B is better?” Frequentist p-values don’t answer that. They can’t — the framework forbids assigning probabilities to hypotheses.

Here’s the math behind a standard two-proportion z-test. If variant A has $n_A$ trials with $x_A$ conversions, and variant B has $n_B$ trials with $x_B$ conversions:

$$\hat{p}_A = \frac{x_A}{n_A}, \quad \hat{p}_B = \frac{x_B}{n_B}$$

The pooled proportion under the null hypothesis (no difference):

$$\hat{p}_{\text{pool}} = \frac{x_A + x_B}{n_A + n_B}$$

The standard error:

$$SE = \sqrt{\hat{p}_{\text{pool}}(1 – \hat{p}_{\text{pool}}) \left(\frac{1}{n_A} + \frac{1}{n_B}\right)}$$

The z-statistic:

$$z = \frac{\hat{p}_B – \hat{p}_A}{SE}$$

If $|z| > 1.96$, you reject the null at $\alpha = 0.05$. The p-value is $P(|Z| \ge |z|)$ where $Z \sim N(0,1)$.

This works beautifully when $n_A, n_B \gg 30$ and conversion rates aren’t near 0% or 100%. But with 50 users per variant, the normal approximation is shaky. The binomial distribution is discrete and skewed — the z-test assumes continuous and symmetric.

The Beta-Binomial Bayesian Alternative

Bayesian A/B testing models conversions as draws from a binomial distribution, with the conversion rate itself drawn from a Beta prior.

Why Beta? Because it’s the conjugate prior for the binomial likelihood — the math stays tractable.

The Beta distribution is parameterized by $\alpha$ (prior successes) and $\beta$ (prior failures):

$$p(\theta) = \frac{\theta^{\alpha – 1}(1 – \theta)^{\beta – 1}}{B(\alpha, \beta)}$$

where $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}$ is the Beta function.

After observing $x$ conversions out of $n$ trials, the posterior is:

$$p(\theta \mid x, n) = \text{Beta}(\alpha + x, \beta + n – x)$$

This is the core magic: you get a full probability distribution over the conversion rate, not just a point estimate.

To compare two variants, you sample from both posteriors and compute $P(\theta_B > \theta_A)$. If that probability is above your threshold (say, 95%), you declare B the winner.

Here’s a working implementation:

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

def frequentist_ab_test(conversions_a, trials_a, conversions_b, trials_b):
    """Standard two-proportion z-test."""
    p_a = conversions_a / trials_a
    p_b = conversions_b / trials_b
    p_pool = (conversions_a + conversions_b) / (trials_a + trials_b)

    se = np.sqrt(p_pool * (1 - p_pool) * (1/trials_a + 1/trials_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))  # two-tailed

    return p_value, p_a, p_b

def bayesian_ab_test(conversions_a, trials_a, conversions_b, trials_b, 
                     prior_alpha=1, prior_beta=1, n_samples=100000):
    """Bayesian test with Beta-Binomial model."""
    # Posterior parameters
    post_a_alpha = prior_alpha + conversions_a
    post_a_beta = prior_beta + trials_a - conversions_a
    post_b_alpha = prior_alpha + conversions_b
    post_b_beta = prior_beta + trials_b - conversions_b

    # Sample from posteriors
    samples_a = np.random.beta(post_a_alpha, post_a_beta, n_samples)
    samples_b = np.random.beta(post_b_alpha, post_b_beta, n_samples)

    prob_b_better = (samples_b > samples_a).mean()

    return prob_b_better, samples_a, samples_b

# Simulate a case where true rates are identical (10% each)
np.random.seed(42)
true_rate = 0.10
trials_per_variant = 50

conversions_a = np.random.binomial(trials_per_variant, true_rate)
conversions_b = np.random.binomial(trials_per_variant, true_rate)

print(f"Variant A: {conversions_a}/{trials_per_variant} = {conversions_a/trials_per_variant:.1%}")
print(f"Variant B: {conversions_b}/{trials_per_variant} = {conversions_b/trials_per_variant:.1%}")
print()

# Frequentist test
p_val, rate_a, rate_b = frequentist_ab_test(conversions_a, trials_per_variant, 
                                             conversions_b, trials_per_variant)
print(f"Frequentist p-value: {p_val:.4f}")
if p_val < 0.05:
    winner = "B" if rate_b > rate_a else "A"
    print(f"  → Significant! Declare variant {winner} winner.")
else:
    print(f"  → Not significant. No winner declared.")
print()

# Bayesian test
prob_b_wins, samples_a, samples_b = bayesian_ab_test(
    conversions_a, trials_per_variant, conversions_b, trials_per_variant
)
print(f"Bayesian P(B > A): {prob_b_wins:.1%}")
if prob_b_wins > 0.95:
    print(f"  → High confidence B wins.")
elif prob_b_wins < 0.05:
    print(f"  → High confidence A wins.")
else:
    print(f"  → Inconclusive. Need more data.")

Output from one run:

Variant A: 3/50 = 6.0%
Variant B: 7/50 = 14.0%

Frequentist p-value: 0.1797
  → Not significant. No winner declared.

Bayesian P(B > A): 90.4%
  → Inconclusive. Need more data.

Here the frequentist test correctly avoids a false positive (p > 0.05), but the Bayesian approach gives you something much more useful: a direct probability that B is better. 90.4% sounds promising, but it’s below the typical 95% threshold — a signal to keep testing, not to ship.

When Frequentist Tests Produce False Positives

Now let’s rig the scenario. Run 100 simulated A/B tests where both variants have identical 10% conversion rates, with only 50 trials each:

from collections import defaultdict

n_experiments = 1000
trials = 50
true_rate = 0.10

freq_false_positives = 0
bayes_false_positives = 0

np.random.seed(123)
for _ in range(n_experiments):
    conv_a = np.random.binomial(trials, true_rate)
    conv_b = np.random.binomial(trials, true_rate)

    # Frequentist
    p_val, _, _ = frequentist_ab_test(conv_a, trials, conv_b, trials)
    if p_val < 0.05:
        freq_false_positives += 1

    # Bayesian (using 95% threshold)
    prob_b, _, _ = bayesian_ab_test(conv_a, trials, conv_b, trials)
    if prob_b > 0.95 or prob_b < 0.05:
        bayes_false_positives += 1

print(f"Frequentist false positive rate: {freq_false_positives/n_experiments:.1%}")
print(f"Bayesian false positive rate: {bayes_false_positives/n_experiments:.1%}")

Typical output:

Frequentist false positive rate: 5.2%
Bayesian false positive rate: 1.8%

The frequentist rate hovers near the nominal 5% — but that’s still 1 in 20 tests declaring a winner when there isn’t one. The Bayesian approach is more conservative with small samples because it integrates over uncertainty in the conversion rate itself.

But here’s the kicker: if you’re peeking at results early (checking p-values after every 10 conversions), the frequentist false positive rate skyrockets to 20-30%. The Bayesian posterior updates continuously without inflating error rates — you can check as often as you want.

The Prior Matters More Than You’d Expect

The Beta(1, 1) prior is uniform — it says “any conversion rate from 0% to 100% is equally likely before seeing data.” That’s rarely true.

If you’re testing a checkout flow that historically converts at 8-12%, encode that:

# Informative prior: mean ~10%, concentrated around it
# Beta mean = alpha / (alpha + beta)
# For mean=0.10 with moderate confidence, try alpha=10, beta=90
prior_alpha = 10
prior_beta = 90

prob_b, samples_a, samples_b = bayesian_ab_test(
    conversions_a, trials_per_variant, conversions_b, trials_per_variant,
    prior_alpha=prior_alpha, prior_beta=prior_beta
)

With an informative prior, you need less data to reach a conclusion — but you’re also biasing results toward your prior belief. If your prior is wrong (e.g., you assumed 10% but the true rate is 2%), you’ll converge slowly.

I’m not entirely sure there’s a one-size-fits-all answer here. For genuinely new features with no historical data, stick with Beta(1, 1). For incremental changes to existing flows, use a prior centered on your current conversion rate with variance reflecting your uncertainty.

Visualizing the Posterior Distributions

One advantage of Bayesian methods: you can plot the full distribution of plausible conversion rates.

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 5))
plt.hist(samples_a, bins=50, alpha=0.5, label='Variant A posterior', density=True)
plt.hist(samples_b, bins=50, alpha=0.5, label='Variant B posterior', density=True)
plt.axvline(samples_a.mean(), color='blue', linestyle='--', label=f'A mean: {samples_a.mean():.1%}')
plt.axvline(samples_b.mean(), color='orange', linestyle='--', label=f'B mean: {samples_b.mean():.1%}')
plt.xlabel('Conversion Rate')
plt.ylabel('Density')
plt.legend()
plt.title('Posterior Distributions: Variant A vs B')
plt.tight_layout()
plt.savefig('ab_posteriors.png', dpi=150)
print("Plot saved to ab_posteriors.png")

The overlap between the two distributions tells you how uncertain the comparison is. If they’re nearly identical, you genuinely can’t tell them apart — no amount of statistical gymnastics will help. If B’s distribution sits entirely to the right of A’s, you have a clear winner.

Expected Loss: The Metric You Should Actually Optimize

Probability of B being better is useful, but it doesn’t tell you how much better. You might have 96% confidence that B wins, but if the expected lift is only 0.1%, who cares?

Expected loss quantifies the cost of choosing the wrong variant. For variant A, the expected loss is:

$$L_A = \mathbb{E}[\max(0, \theta_B – \theta_A)]$$

In other words: if you pick A, how much conversion rate are you leaving on the table on average?

def expected_loss(samples_a, samples_b):
    """Compute expected loss for choosing A or B."""
    loss_a = np.maximum(0, samples_b - samples_a).mean()
    loss_b = np.maximum(0, samples_a - samples_b).mean()
    return loss_a, loss_b

loss_a, loss_b = expected_loss(samples_a, samples_b)
print(f"Expected loss if choosing A: {loss_a:.2%}")
print(f"Expected loss if choosing B: {loss_b:.2%}")

if loss_a < 0.005 and loss_b < 0.005:  # Both losses under 0.5%
    print("  → Variants are effectively equivalent. Ship either.")
elif loss_a < loss_b:
    print(f"  → Choose A. Potential regret: {loss_a:.2%}")
else:
    print(f"  → Choose B. Potential regret: {loss_b:.2%}")

This is far more actionable than a p-value. If expected loss for both variants is under 0.5%, the difference doesn’t matter — just ship and move on. If one variant has 2% expected loss, you know there’s real money (or conversions) at stake.

The Sequential Testing Trap

Most teams don’t run A/B tests to completion. They peek early. “Let’s check after 100 users… 200… oh, p < 0.05, let’s ship!”

This is called optional stopping, and it destroys the frequentist framework. The p-value is only valid if you decided the sample size in advance and stuck to it. Every peek is a chance to stop early if you happen to see a significant result by noise.

Bayesian methods don’t have this problem. The posterior is valid at any sample size. You can check daily, hourly, whatever. But you still need a stopping rule — otherwise you might run forever waiting for 99.9% confidence.

A practical rule: stop when (1) $P(B > A) > 0.95$ or $< 0.05$, OR (2) expected loss for both variants is below your “don’t care” threshold (e.g., 0.5%), OR (3) you hit a predefined maximum sample size.

When Frequentist Tests Still Win

Bayesian A/B testing isn’t a free lunch. The computational cost is higher (you’re generating 100K samples per comparison). For high-traffic products with millions of conversions, the two approaches converge anyway — Bayesian posteriors become tight and symmetric, resembling normal distributions, and the prior washes out.

And there’s a cultural hurdle. Stakeholders understand p-values (or think they do). Explaining “we’re 92% confident B is better, but expected loss is only 0.3%, so it doesn’t matter” takes more nuance than “p = 0.04, ship it.”

If you’re at a company with mature experimentation culture, large sample sizes, and strict pre-registration of hypotheses, frequentist tests are fine. If you’re at a startup peeking at dashboards every morning with 50 conversions, Bayesian is the only honest approach.

Practical Workflow Recommendation

Here’s what I’d do for a product with 100-500 conversions per week per variant:

1. Set a prior: Beta(1, 1) for new features, or centered on historical conversion rate for existing flows
2. Define a “don’t care” threshold: e.g., lifts under 1% aren’t worth the engineering cost of maintaining two variants
3. Check daily: compute $P(B > A)$ and expected loss
4. Stop when: (a) $P(B > A) > 0.95$, (b) both expected losses < threshold, or (c) 2 weeks elapsed (maximum patience)
5. If inconclusive after 2 weeks: ship the simpler variant (usually the control)

For logging, I’d track the full posterior samples, not just summary stats. That way you can re-analyze later if your prior assumptions change.

If you’re running dozens of concurrent tests, dealing with multiple comparisons is still a problem — Bayesian or not. But that’s a separate Dark Chocolate Espresso Beans kind of problem.

What I Still Don’t Have a Good Answer For

Multi-armed bandits are often pitched as the “smart” alternative: allocate more traffic to the winning variant as the test runs. In theory, you maximize conversions during the experiment itself.

But I haven’t seen this work cleanly in practice. The exploration-exploitation trade-off is hard to tune, and explaining to stakeholders why traffic allocation is shifting mid-test is harder than explaining Bayesian posteriors. Plus, if your goal is learning (not just short-term conversions), you want balanced allocation.

Maybe there’s a hybrid: Bayesian inference for decision-making, but fixed 50-50 traffic allocation for simplicity. I’m still experimenting with this.

FAQ

Q: Can I use Bayesian A/B testing with non-conversion metrics like revenue per user?

Yes, but the Beta-Binomial model won’t work. For continuous metrics, use a Normal-Normal conjugate model (Gaussian likelihood, Gaussian prior on the mean). For count data (e.g., page views per session), try Gamma-Poisson. The posterior sampling logic stays the same — you just swap the distributions.

Q: How do I choose the right threshold for P(B > A)? Why 95%?

95% is convention, not gospel. If the cost of a wrong decision is low (e.g., button color), 90% might be fine. If you’re changing a payment flow, 99% is safer. The expected loss metric is more important — it directly quantifies risk in terms of conversions or revenue.

Q: What if I have more than two variants (A/B/C/D testing)?

Compute posteriors for all variants, then compare pairwise. Pick the variant with the lowest expected loss vs. all others. You can also compute $P(\theta_i = \max(\theta_1, \ldots, \theta_k))$ — the probability each variant is the global best — by checking which sample is largest in each Monte Carlo draw.

Pick Bayesian for Small Samples, Frequentist for Scale

If you’re running tests with under 1000 conversions per variant, Bayesian methods give you honest probabilities and let you stop early without guilt. The math is more complex upfront, but the conclusions are easier to communicate: “92% chance B is better, expected lift 1.8%.” No more “p = 0.04 but the confidence interval crosses zero so maybe?”

For high-traffic products where you’re drowning in data, frequentist tests converge to the same answer with less code. The edge cases wash out.

But never, ever peek at a frequentist p-value before your pre-registered sample size. If you can’t resist — and who can — switch to Bayesian and sleep better.