Advanced Experiment Design — Power Analysis Quick Guide

Why power matters (and what this guide helps you do)

Power analysis helps you plan experiments that produce decision-ready evidence. Underpowered tests create noisy, misleading results; overpowered tests waste time and money. This guide gives practical rules, simple formulas, worked examples, and a short decision flow to help you choose sample sizes and minimal detectable effects (MDE) for common experiment types: mean differences, proportion differences, clustered designs, and simple factorials. Pseudocode is included so you can build a quick calculator or spreadsheet.

Core concepts (plain language)

• Effect size (δ). The smallest difference you care about detecting with reasonable confidence (absolute or relative).
• Alpha (α). Type I error rate (commonly 0.05). The probability you’ll falsely claim an effect when none exists.
• Power (1−β). Probability of detecting the true effect of size δ (commonly targeted at 0.8 or 0.9).
• Baseline variability (σ or p). Standard deviation for continuous outcomes or baseline proportion for binary outcomes—key drivers of sample size.
• Minimal Detectable Effect (MDE). The smallest effect your design can detect with chosen α, power, and sample size.

Simple two-group mean test (equal allocation)

Use this when outcomes are approximately continuous and randomization is at the individual level.

Approximate per-group sample size (large-sample normal approximation):

n per group ≈ 2 × (Z_{1−α/2} + Z_{1−β})^2 × σ^2 / δ^2

Where Z_{q} is the q-quantile of the standard normal distribution (e.g., Z_{0.975}=1.96, Z_{0.84}≈0.994).

Worked example

Baseline standard deviation σ=10 units, target δ=3 units, α=0.05, power=0.8.

Z_{0.975}=1.96, Z_{0.8}=0.84 → n ≈ 2 × (1.96+0.84)^2 × 100 / 9 ≈ 2 × (2.8)^2 × 11.11 ≈ 2 × 7.84 × 11.11 ≈ 174.3 → about 175 per group.

Two-proportion test (comparing conversion rates)

Approximate combined sample size (equal allocation):

n per group ≈ (Z_{1−α/2} + Z_{1−β})^2 × [p1(1−p1)+p2(1−p2)] / (p1−p2)^2

Often use p1≈baseline p0 and p2=p0+δ.

Worked example

Baseline p0=0.10, want to detect δ=0.02 (absolute), α=0.05, power=0.8.

p1=0.10, p2=0.12 → variance term ≈ 0.10×0.90 + 0.12×0.88 = 0.09 + 0.1056 = 0.1956. Zs ≈ 2.8.

n ≈ (2.8)^2 × 0.1956 / 0.0004 ≈ 7.84 × 489 → ≈ 3,835 per group.

Adjusting for clustering (cluster- or group-randomized)

When randomization/unit is a cluster (class, clinic, store), account for the intra-class correlation (ICC). Effective sample size is reduced. Design effect (DE):

DE = 1 + (m − 1) × ICC

Where m is average cluster size. Multiply the per-group sample size by DE to get required subjects, or divide number of clusters accordingly. For clustered proportions and counts, replace σ^2 with the appropriate between/within variance structure.

Factorial designs and split-plot notes

Factorial designs test multiple factors simultaneously. Power to detect an interaction usually requires larger sample sizes than main effects. When factors are orthogonal and balanced, sample size formulas for each main effect resemble the two-group formulas but use pooled variance and appropriate allocation fractions. If randomization occurs at different levels (split-plot), compute power per randomization unit and treat each contrast on its correct unit.

Quick pseudocode (two-sample mean)

    INPUT: alpha, power, sigma, delta
    z_alpha = NormalQuantile(1 - alpha/2)
    z_beta = NormalQuantile(power)
    n = 2 * (z_alpha + z_beta)^2 * sigma^2 / delta^2
    RETURN ceil(n)
  

Practical guidance for baseline variability

• If you have historical data, estimate σ from recent, comparable populations and time windows.
• When historical stability is low, plan conservatively (use larger σ percentile) or run a short pilot to measure variability.
• For proportions near 0 or 1, variance is lower, but relative MDE (relative to baseline) can be large—report both absolute and relative MDE.

Tradeoffs: MDE vs. runtime and cost

MDE is inversely proportional to sqrt(n). Halving the MDE requires roughly quadrupling sample size. Use this to map business costs: how many days/customers/units do you need to run to reach desired MDE? If runtime is infeasible, consider:

• Accepting a larger MDE that still matters to stakeholders.
• Improving measurement precision (reduce σ) via better instrumentation or more consistent cohorts.
• Using a more sensitive metric aligned with your hypothesis.
• Using sequential/adaptive designs (see next section).

When to consider sequential or adaptive designs

Sequential or group-sequential designs allow interim looks with adjusted significance thresholds, potentially shortening runtime. Consider them when:

• Expected effect size is small and long fixed-horizon runs are costly.
• There are ethical or business reasons to stop early for harm or benefit.
• Outcome accrual is slow and you want early signals.

Sequential methods require pre-specification of stopping rules and appropriate statistical corrections (e.g., O'Brien–Fleming, Pocock). If you lack experience, consult a statistician—misusing sequential rules inflates Type I error or yields biased estimates.

Practical decision flow (short)

1. Define the outcome and business-relevant minimum effect (δ).
2. Estimate baseline variance/proportion from recent data or pilot.
3. Choose α and target power (commonly 0.05 and 0.8 or 0.9).
4. Compute n with the appropriate formula and adjust for clustering/design.
5. Translate n into expected runtime or number of units; check feasibility.
6. If infeasible, consider (a) larger δ, (b) better measurement, (c) alternative metrics, or (d) sequential/adaptive design with expert input.

Checklist before you run the experiment

• Have you identified the correct experimental unit and clustered structure?
• Did you base σ or p on comparable historical data or a pilot?
• Is your MDE aligned with stakeholder value and decision thresholds?
• Have you pre-registered primary metric, analysis plan, and stopping rules (if sequential)?
• Do you have rollout and exclusion rules to avoid post-hoc filtering?

Next practical steps

1) Build a simple calculator (spreadsheet or script) using the pseudocode above. 2) If you need saved inputs, team sharing, or automated reports, consider turning the calculator into an interactive form that stores submissions. 3) For clustered, factorial or split-plot designs or when ethics/regulation apply, consult a statistician before finalizing the plan.

References and further reading

• Statistical texts on sample size and sequential analysis (standard references).
• Practical experiment design guides for product and clinical contexts.
• Platform-specific calculators and libraries (for engineers: R's pwr package, Python's statsmodels).

Note: This guide focuses on practical design and approximations. It does not replace domain-specific ethics, regulatory review, or professional statistical consulting when required.