A/B Test Planner & Power Calculator

Welcome — what this tool helps you do

This planner helps teams move from a vague idea to a well-specified, adequately powered A/B test that yields defensible, actionable results. Use it to frame the business question, pick a clear primary metric, calculate a realistic sample size, and pre-specify analysis and stopping rules so your conclusions are credible.

Why this matters (the core hunger)

Too many experiments are underpowered, ambiguous about metrics, or open to selective reporting. That wastes time and can lead to poor decisions. A good A/B plan protects you from false positives, false negatives, and operational surprises by making assumptions explicit and ensuring you have enough data to detect the changes you actually care about.

Step-by-step planner (fill these before running the test)

1. Business question: What decision will this experiment support? (Example: Should we expose customers to promotion X to increase sign-ups?)
2. Primary metric: One clear, pre‑specified metric that maps to the business decision (e.g., sign-up rate within 7 days). Avoid multiple primary metrics.
3. Baseline: Historical rate or mean for the primary metric (include calculation window and segmentation). Document sample size and variability estimates that produced this baseline.
4. Minimum Detectable Effect (MDE): The smallest change (absolute or relative) that would change your decision. Be explicit whether MDE is absolute (e.g., +1 percentage point) or relative (e.g., +10% uplift).
5. Statistical criteria: Alpha (commonly 0.05 two-sided), desired power (commonly 80% or 90%), and whether the test is one- or two-sided.
6. Allocation and randomization plan: Unit of randomization (user, session, account), assignment method, and how you’ll ensure balance and avoid cross-contamination.
7. Guardrail metrics: Secondary metrics to detect harms (e.g., engagement, revenue per user, error rate). Pre-specify thresholds that would abort the rollout even if the primary metric moves favorably.
8. Analysis plan: Exact estimator (e.g., difference in proportions, logistic regression with covariates), inclusion/exclusion rules, handling of missing data, and adjustment for multiple comparisons if you will test multiple variants or segments.
9. Stopping rules: Whether you will use fixed-horizon inference (recommended) or pre-defined sequential monitoring with an alpha spending plan. Avoid ad-hoc peeking unless you pre-specify corrections.
10. Rollout strategy: Criteria for promotion to production, rollout percentage steps, and monitoring plan post-rollout (validate external validity and detect regression drift).

Sample-size guidance (how to think about power)

Two common cases cover most A/B tests: comparing two proportions (binary outcomes like conversion) and comparing two means (continuous outcomes like revenue per visit).

Quick formulas (approximate)

Two means (equal allocation):
n per group ≈ 2 × (Z_{1-α/2} + Z_{1-β})^2 × σ^2 / Δ^2
where σ is the standard deviation and Δ is the minimum detectable difference (absolute).

Two proportions (normal approximation, equal allocation):
n per group ≈ [Z_{1-α/2} √(2 p̄(1−p̄)) + Z_{1-β} √(p1(1−p1)+p2(1−p2))]^2 / (p1−p2)^2
where p1 is baseline, p2 = p1 + Δ (absolute), and p̄=(p1+p2)/2. Use these as planning approximations; software uses the exact formulas.

Interpretation tips

• Smaller MDEs require much larger samples. Expect rapidly increasing sample needs as MDE shrinks.
• Baseline rate matters: detecting the same absolute change is harder when the baseline is near 50% (higher variance) than when it’s near 0 or 1.
• If the required sample is impractically large, either raise the MDE (accept only bigger, more actionable changes), improve measurement precision (reduce σ), or consider alternative designs.

Reference sample-size table (approximate, two-sided α=0.05, power=80%)

Approximate samples per group for common baselines and absolute MDEs:

Note: These numbers are approximate and intended to communicate scale. Compute exact sample sizes using statistical software tied to your pre-specified metric, allocation, and test design.

Example analysis scripts

R (two-sample proportion)

power.prop.test(p1 = 0.05, p2 = 0.06, power = 0.8, sig.level = 0.05, alternative = "two.sided")
# Returns total and per-group sample size (approximate)

Python (statsmodels)

from statsmodels.stats.proportion import proportion_effectsize
from statsmodels.stats.power import NormalIndPower

p1 = 0.05
p2 = 0.06
effect = proportion_effectsize(p1, p2)
analysis = NormalIndPower()
n_per_group = analysis.solve_power(effect_size=effect, power=0.8, alpha=0.05, ratio=1)
print(n_per_group)

For continuous outcomes use statsmodels.stats.power.TTestIndPower or the equivalent R functions. If you plan covariate adjustment (e.g., regression), document it and consider smaller effective sample sizes due to variance reduction.

Common pitfalls & how to avoid them

• Underpowered tests: Don’t run experiments that can’t detect practically important effects. If sample size is too small, consider changing the question or measurement strategy.
• Peeking: Repeatedly looking at results without proper sequential methods inflates false positives. Prefer fixed-horizon analysis or pre-specified sequential monitoring (with alpha spending).
• Vague metrics: Avoid composite or loosely defined metrics. Pre-specify calculation windows, denominators, and inclusion rules.
• Selective reporting: Pre-register the primary metric and analysis plan. Report guardrails and all pre-specified secondary analyses.
• Ignoring implementation: Randomization failure, logging errors, or cross-device issues can bias results. Include randomization checks and instrumentation tests in your plan.

Pre-launch checklist (quick copyable checklist)

1. Business question and decision rule documented
2. Primary metric defined and baseline computed
3. MDE justified and sample size estimated
4. Randomization unit and implementation plan verified
5. Guardrail metrics selected
6. Analysis plan pre-specified and saved (pre-registration)
7. Stopping rules and monitoring cadence defined
8. Rollout and rollback procedures documented

Next steps & recommended experiment discipline

Pre-register the plan (metric, analysis, sample size, stopping rules). Run a short pilot if the instrumentation is new. After the test, validate assumptions (check balance, check for coding errors), run the pre-specified analysis, and prepare an operational decision memo that maps statistical outcomes to the operational action you will take.

Capability enhancements (suggested)

This content would be meaningfully improved by adding:

• An interactive sample-size calculator that accepts baseline, MDE (absolute or relative), α, power, allocation ratio, and outputs per-group and total sample sizes and estimated duration given traffic — ideally implemented client-side or by a callable service (avoid server-side black boxes without auditability).
• A pre-registration form (interactive) that stores trial parameters (using the platform's content submission capability) so experiments are auditable and reproducible.
• Integration with experiment-logging or analytics systems to automatically estimate baseline and expected duration from real traffic data.

References & further reading

• Books and guides on experimentation and causal inference (recommend your organization’s preferred references).
• Statsmodels and R documentation for power/sample-size functions.