Power Analysis Quick Guide

Why power analysis matters (and what it won't do for you)

Power analysis helps you choose a sample size that gives a good chance of detecting an effect of practical interest while avoiding wasted resources. It connects four things you care about: the effect size you want to detect, the acceptable false-positive rate (alpha), the desired probability of detection (power), and the variability or noise in your measurements. Done well, it improves interpretability, reproducibility, and resource allocation. Done poorly, it creates false confidence or wastes time.

Core concepts (plain language)

• Effect size — the magnitude of the difference or relationship you care about. Think in practical terms: what difference would change a decision or belief?
• Alpha (α) — the tolerated probability of a false positive (commonly 0.05). Lower alpha requires larger samples for the same power.
• Power (1 − β) — the probability of correctly detecting the effect size you specified (commonly 0.8 or 0.9). Higher power needs more data.
• Variability — how noisy your measurements are (standard deviation or within-group variance). More variability increases required sample size.
• One-sided vs two-sided tests — directional hypotheses (one-sided) need slightly fewer samples than two-sided tests.
• Multiple comparisons and design choices — ANOVA, repeated measures, covariates, blocking, or clustered designs change calculations and usually require adjustments.

Quick heuristics

• Small effect (hard to detect): expect hundreds per group.
• Medium effect (practically meaningful in many contexts): tens per group (order 30–70).
• Large effect (obvious differences): low double-digit or fewer per group.
• If you have limited resources, prioritize reducing measurement variability (better protocols, blocking, paired/repeated measures) — this often reduces required sample size more than fiddling with alpha.

Worked examples (step-by-step)

1) Two-sample t-test (equal groups, two-sided)

Goal: detect a standardized effect size Cohen's d = 0.5 (a common “medium” benchmark), with α = 0.05 and power = 0.8.

Approximate formula for balanced groups (useful quick estimate):

n per group ≈ 2 × (Z_{α/2} + Z_{β})^2 / d^2

Where Z_{α/2} ≈ 1.96 (for α = 0.05) and Z_{β} ≈ 0.84 (for power = 0.8). Plugging in:

n ≈ 2 × (1.96 + 0.84)^2 / 0.5^2 ≈ 2 × 7.84 / 0.25 ≈ 63 per group.

Interpretation: about 60–65 subjects per group are needed. For a quick planning stage, round up and consider 70 per group to allow for dropouts or higher variability.

2) One-way ANOVA (three groups, medium effect)

Goal: detect a medium ANOVA effect (Cohen’s f ≈ 0.25) with α = 0.05 and power = 0.8. ANOVA power depends on number of groups and balance; for a balanced 3-group design this commonly yields around 50–55 subjects per group.

Practical takeaway: expect roughly 150–170 total subjects (≈50–60 per group) for a medium-sized group effect. Use a dedicated tool (G*Power, R pwr) for the precise calculation that matches your planned contrasts and multiple-comparison adjustments.

3) Difference in proportions (binary outcome)

Goal: detect a difference from p1 = 0.20 to p2 = 0.35 (an absolute difference 0.15), α = 0.05, power = 0.8.

Plugging the standard two-proportion formula (pooled approximation) yields an estimated ≈ 130–140 subjects per group (calculation example gives ~138 per group). Binary outcomes often require larger samples than continuous outcomes for the same effect size.

Practical tips and good practice

• Define a meaningful effect before collecting data. Power analysis should reflect an effect size that would change decisions, not just the smallest detectable difference possible.
• Use variance estimates from pilot data or past studies. If you don't have data, run a small pilot to estimate variability and then update your power calculation.
• Consider precision rather than power alone. For estimation-focused work, choose an N that gives an acceptably narrow confidence interval for the effect size.
• Pre-register your analysis plan and sample-size rationale. This reduces selective reporting and increases credibility.
• Adaptive and sequential designs can save resources. Group sequential methods and pre-planned interim checks allow stopping early for efficacy or futility — learn the method and account for alpha spending.
• Reduce unnecessary variability. Use blocking, pairing, covariates, or within-subject designs when appropriate — they often reduce N more than changing α or power.
• Correct for clustering and multiple comparisons. Clustered or hierarchical data require design-effect adjustments; multiple endpoints or many pairwise tests need correction or an explicit multiplicity plan.
• Report all choices transparently. Include the effect size, alpha, power, variance assumptions, software or formula used, and any adjustments in methods and publications.

Common mistakes to avoid

1. Planning power based on the observed effect size from the same (small) pilot — this leads to optimistic underpowered designs.
2. Using arbitrary sample sizes without connecting them to effect size or variability.
3. Failing to adjust for expected dropouts, missing data, or cluster design effects.
4. Neglecting multiplicity when testing several endpoints or pairwise comparisons.

Where to compute precisely (recommended tools)

• G*Power (free standalone GUI) — flexible for t-tests, ANOVA, proportions, correlations.
• R pwr package — functions like pwr.t.test(), pwr.anova.test(), pwr.2p.test().
• Python statsmodels (statsmodels.stats.power) — scripts for reproducible pipelines and simulations.
• Online calculators from trusted academic sites — use them for quick checks, but prefer reproducible code for final plans.

Checklist before you finalize sample size

• Have you specified a practically meaningful effect size?
• Are your variability estimates realistic (pilot or literature-based)?
• Have you accounted for dropouts, missing data, and cluster effects?
• Does your design (paired, repeated measures, blocking) reduce variability in planned ways?
• Have you planned multiplicity adjustments or pre-specified primary endpoint(s)?
• Is your analysis plan and sample-size rationale documented and ready to pre-register?

Next practical steps

Run several scenarios across likely effect sizes and variances (a short "sensitivity sweep"), choose a defensible N or an adaptive plan, and document the choices. If resources are very limited, invest in better measurements, blocking, or a small pilot instead of assuming a tiny N will provide convincing evidence.

References and further reading

• G*Power manual and downloads
• R package pwr documentation
• Textbooks and tutorials on sample size and sequential designs