Power Analysis Quick Guide

Why this matters

Teams run experiments to learn what really works. Power analysis helps you answer a simple operational question: how much data do we need so a real effect of a given size is unlikely to be missed? Getting this right reduces wasted experiments, avoids false confidence, and helps you prioritize ideas that are worth the cost of testing.

What you'll learn

• Key ideas: power, alpha, beta, minimum detectable effect (MDE), and sample size.
• Quick formulas and two worked examples (proportions and continuous metrics).
• Practical heuristics when data is limited.
• An annotated spreadsheet blueprint you can copy and adapt.

Key concepts (plain language)

Alpha (α) — the allowed false-positive risk (commonly 0.05). If you use α=0.05, you accept a 5% chance of declaring an effect when none exists.

Power (1−β) — the probability your test will find an effect of the size you care about, when it actually exists. Common targets are 80% or 90%.

Minimum Detectable Effect (MDE) — the smallest true change you consider practically important and want the experiment to have a good chance of detecting.

Sample size — the number of units (users, sessions, days, clusters) required to reach the targeted power for the chosen MDE and α.

Simple rules of thumb

• Pick MDE based on business value, not just detectability. A very small MDE may be statistically detectable but useless in practice.
• When baseline rates are low (rare events), sample sizes grow quickly to detect small absolute differences.
• For continuous metrics, express MDE as a fraction of the standard deviation (Cohen’s d): 0.2 small, 0.5 medium, 0.8 large.
• Default practical targets: α = 0.05 (two-sided), power = 0.80. Raise power to 90% for high-cost decisions.
• If you have clustering (e.g., users within schools or days), inflate sample sizes with a design effect (see below).

Quick formulas and two worked examples

1) Two-sample proportions (common for conversion/A-B tests)

A commonly used approximate formula for per-group sample size is:

n ≈ [ (z_{1−α/2}·√(2·p̄·(1−p̄)) + z_{1−β}·√(p1·(1−p1) + p2·(1−p2)) )² ] / (p2 − p1)²

Where p1 is baseline, p2 is expected variant rate, p̄=(p1+p2)/2, z values are normal quantiles (e.g., 1.96 for α=0.05, two-sided; 0.84 for 80% power).

Example: baseline p1 = 10% (0.10). You want to detect an absolute lift of 1 percentage point (p2 = 0.11). With α=0.05, power=0.80:

• p̄ = 0.105
• z_{1−α/2} = 1.96, z_{1−β} = 0.84
• Compute the numerator: (1.96·√(2·0.105·0.895) + 0.84·√(0.10·0.90 + 0.11·0.89)) ≈ 1.213
• Square and divide by Δ² = 0.01² = 0.0001 → n ≈ 14,710 per group (≈29,420 total)

Interpretation: detecting a 1 percentage-point absolute lift on a 10% baseline requires a very large sample. If that sample is impractical, consider increasing the MDE (test larger changes), running a longer test, or using a different metric with lower variance.

2) Continuous outcome (difference in means)

For standardized effect-size d = Δ / σ (difference divided by standard deviation), a useful approximate formula for equal group sizes is:

n per group ≈ 2·(z_{1−α/2} + z_{1−β})² / d²

Example: you want to detect a 0.2σ difference (small effect) with α=0.05, power=0.80. z-sum = 1.96 + 0.84 = 2.80.

• n ≈ 2·(2.80)² / 0.2² = 15.68 / 0.04 ≈ 392 per group (≈784 total)

So detecting a 0.2 standard-deviation change usually requires hundreds of observations per arm.

Clustered designs and design effect

When randomization or measurement happens at a clustered level (e.g., classrooms, stores, days), effective sample size is reduced. Use the design effect:

DE = 1 + (m − 1)·ICC

Where m is average cluster size and ICC is the intraclass correlation coefficient (similarity within cluster). Adjusted sample = nominal sample · DE.

Example: m = 50, ICC = 0.02 → DE = 1 + 49·0.02 = 1.98 → nearly double the nominal sample required.

Practical shortcuts and constrained environments

• Prefer business-significant MDEs. Ask: “What minimum change would change our decision or revenue?”
• If you can’t reach the sample for that MDE, run a pilot to estimate variance and inform a later powered test. Use pilot data only to estimate variance—not to claim significance.
• Use sequential or group-sequential designs carefully (and pre-specify stopping rules) to avoid inflated false-positive rates.
• If multiple metrics or comparisons are planned, adjust α (or control false discovery rate) and increase sample accordingly.
• For rare events, consider using more efficient metrics (e.g., aggregated rates over longer windows, composite metrics) or focused experiments on higher-intensity segments.

Checklist: When to run power analysis vs. pilot

1. If the decision based on the experiment is high-impact (pricing changes, clinical launches, product-wide changes) → run full power analysis and design for 80–90% power.
2. If variance or ICC is unknown and you have limited traffic → run a short pilot to estimate variability, then compute powered sample size.
3. If the expected effect size is tiny and sample costs are high → re-evaluate whether the test is worth running or change the MDE to a value that would change decisions.
4. If you plan many exploratory comparisons → pre-register primary outcome and adjust for multiplicity or treat other outcomes as hypothesis-generating.

Annotated spreadsheet template (what to include)

Copy these columns into a sheet so product or research teams can enter their numbers:

1. MetricType: "proportion" or "mean"
2. Baseline (p or mean)
3. SD (for means) or alternative metric inputs
4. Target/MDE (absolute or SD units)
5. Alpha (0.05 default)
6. Power (0.80 or 0.90)
7. One- or two-sided test
8. Calculated z values (use Excel: =NORM.S.INV(1-Alpha/2) and =NORM.S.INV(Power))
9. Sample size formula cells (examples below)

Example Excel formula for two-sample proportions (per-group), using P1 in A2, P2 in A3, Alpha in A4, Power in A5:

=((NORM.S.INV(1-A4/2)*SQRT(2*((A2+A3)/2)*(1-((A2+A3)/2))) + NORM.S.INV(A5)*SQRT(A2*(1-A2)+A3*(1-A3)))^2) / (A3-A2)^2

Example for standardized difference (d) in Excel, with Alpha in B1 and Power in B2 and d in B3:

=2*(NORM.S.INV(1-B1/2)+NORM.S.INV(B2))^2 / B3^2

Note: adapt these formulas for one-sided tests, unequal group sizes, or cluster adjustments (multiply n by the design effect).

Good practice and risk control

• Pre-specify primary metric, MDE, alpha, and stopping rules before data collection starts.
• Record and report whether sample-size targets were met; interpret underpowered results cautiously.
• Avoid post-hoc hypothesis fishing. If you explore many hypotheses, treat findings as exploratory and follow them up with powered replication tests.
• When in doubt for critical decisions, consult a statistician or methodologist—this guide helps teams make better decisions and know when expert review is needed.

Next practical steps

1. Identify the single most important decision your experiment should influence and choose an MDE tied to that decision.
2. Use the annotated spreadsheet (copy and adapt) or a trusted online calculator to compute required sample size with your metric inputs.
3. If required sample is infeasible, consider increasing MDE, changing the metric, using a longer test window, or moving to a cluster or multi-arm design with appropriate adjustments.
4. Pre-register and run the experiment. Save pilot results and recalibrate future power calculations with updated variance estimates.

Note: This guide provides practical design guidance but does not replace domain-specific ethics, regulatory review, or professional statistical consulting when required.