Reference

Causal Design Quick Reference

A concise, practical reference for choosing and assessing common causal designs when randomized experiments aren’t feasible. Explains each design, key assumptions, typical threats, minimum data needs, quick suitability questions, robustness checks, and a short decision path for when to escalate to a more formal research design.

Purpose

This quick reference helps analysts, team leads, and decision-makers pick the simplest causal approach that can meet evidence standards for your decision. It summarizes common quasi‑experimental and observational designs, their core assumptions, typical threats to validity, minimum data needs, practical checks, and next steps to strengthen inferences.

How to use this reference

Start by stating the causal question (treatment/exposure, outcome, population, timing). Then run the quick suitability questions below and consult the design summaries to select an approach. Pre‑specify outcomes, timing, model features, and robustness checks before looking at results.

Quick suitability checklist

Is randomization feasible within the decision timeframe? If yes, run a randomized experiment.
Do you have longitudinal data covering outcomes before and after treatment for treated and comparison units?
Is there an identifiable cut‑off, threshold, or policy assignment rule you can exploit?
Is treatment assignment plausibly unrelated to potential outcomes after conditioning on observables (selection on observables)?
Can you identify a credible instrument that affects treatment but not the outcome except through treatment?
Are there strong time‑varying confounders or concurrent interventions that could bias before/after comparisons?

Design summaries (when to prefer each)

Randomized Experiment

What: Randomly assign units to treatment and control.

When to prefer: Feasible and ethical; highest internal validity.

Key assumptions: Random assignment ensures balance on observed and unobserved factors.

Threats: Noncompliance, attrition, spillovers, low power.

Minimum data: Pre‑treatment covariates for balance checks, clear outcome measures, sample size/power calculation.

Robustness checks: Balance tests, intention‑to‑treat vs. per‑protocol, varying covariates, subgroup and heterogeneity checks.

Difference‑in‑Differences (DiD)

What: Compare outcome changes over time between treated and comparison groups.

When to prefer: A staggered or one‑time treatment with parallel pre‑treatment trends plausibly holding.

Key assumptions: Parallel trends in the absence of treatment.

Threats: Pre‑trend differences, time‑varying confounders, differential shocks, anticipation effects.

Minimum data: Multiple pre‑ and post‑period observations for both groups.

Checks: Visual pre‑trend plots, placebo (lead) tests, event‑study / dynamic DiD specification, alternative comparison groups.

Interrupted Time Series (ITS)

What: Evaluate a time‑series outcome before and after an intervention at a point in time.

When to prefer: A well‑defined intervention time and many repeated outcome measurements for the same unit.

Key assumptions: No other concurrent interventions or sudden shocks at the intervention time; stability of the trend absent treatment.

Threats: Seasonality, autocorrelation, concurrent policies/events.

Minimum data: Sufficient pre‑intervention observations (rule of thumb: several dozen if possible) and measures to model seasonality and autocorrelation.

Checks: Fit models that account for autocorrelation, include controls for seasonality, check for other events at the same time.

Regression Discontinuity (RD)

What: Exploit a cut‑off in an assignment variable where units just above and below the threshold are compared.

When to prefer: A strict, exogenous threshold determines treatment assignment (e.g., test score cutoff).

Key assumptions: Units cannot precisely manipulate the running variable around the cut‑off; smoothness of potential outcomes in the running variable.

Threats: Manipulation around the cut‑off, discrete running variable, wrong bandwidth choice.

Minimum data: Dense observations around the threshold; running variable measured precisely.

Checks: McCrary or density test, covariate smoothness checks, different bandwidths and polynomial orders, local randomization checks.

Instrumental Variables (IV)

What: Use an instrument that affects treatment uptake but does not affect outcome except through treatment.

When to prefer: Strong endogeneity concerns and a plausible instrument that satisfies relevance and exclusion restrictions.

Key assumptions: Instrument relevance (correlates with treatment) and exclusion (no direct effect on outcome), monotonicity.

Threats: Weak instruments, invalid exclusion, heterogeneous effects complicating interpretation.

Minimum data: Instrument measured for units, strong first‑stage relationship.

Checks: First‑stage F‑statistic, overidentification tests (if >1 instrument), sensitivity analyses, local average treatment effect (LATE) interpretation.

Matching / Propensity Scores

What: Create a comparison group by matching treated units to similar controls on observables.

When to prefer: Treatment selection appears driven primarily by observable covariates and rich covariate data are available.

Key assumptions: Conditional independence / selection on observables; no significant unobserved confounding.

Threats: Hidden bias from unobserved confounders, poor overlap, model dependence.

Minimum data: Rich set of pre‑treatment covariates and sufficient overlap between groups.

Checks: Balance diagnostics, sensitivity (Rosenbaum) bounds, alternative matching algorithms, combined DiD + matching when possible.

Synthetic Control

What: Construct a weighted combination of control units to approximate the treated unit’s pre‑treatment trajectory.

When to prefer: One or a few treated aggregate units (e.g., a state or city) with many potential donor units and long pre‑treatment series.

Key assumptions: Donor pool can produce a close pre‑treatment fit; no spillovers into donor units.

Threats: Poor pre‑treatment fit, inappropriate donor pool, interpolation bias.

Minimum data: Many pre‑treatment periods and a rich donor pool.

Checks: Placebo tests (in-time and in-space), robustness to donor pool restrictions, permutation inference.

Common practical guidance

Pre‑specify your analysis plan: treatment definition, outcomes, timing, covariates, bandwidths, model forms, and robustness checks.
Visualize data early: trends, distributions, balance, and the running variable around cut‑offs.
Use multiple methods where possible: concordant results across approaches strengthen confidence.
Report assumptions and limitations clearly for decision‑makers; describe plausible alternative explanations and effect sizes needed to change conclusions.
Focus on decision thresholds: what effect size would change the decision? Are observed effects practically meaningful or only statistically significant?

Quick robustness checklist (before reporting)

Were pre‑treatment trends inspected and tested?
Were placebo or lead tests performed (no effect before treatment)?
Were alternative model specifications and bandwidths tried?
Were balance and overlap evaluated where matching or weighting were used?
Are results robust to excluding suspicious periods or units?
Was power or detectable minimum effect size assessed?
Are concurrent interventions or shocks considered and documented?

When to escalate to a research design

Consider escalating (more formal study, randomized pilot, or funded evaluation) if one or more of the following apply:

The decision has high stakes (large costs, safety, or strategic impact) and current evidence is weak or biased.
Key design assumptions (parallel trends, exclusion, no manipulation) are implausible or cannot be tested.
Available data are insufficient in frequency, pre‑period length, or measurement quality to support credible inference.
Results are sensitive to reasonable alternative specifications or are driven by a few units.
Pilot randomization or phased rollout is feasible and would meaningfully reduce uncertainty.

Short template: pre‑specification checklist (copyable)

Decision question and policy/practice being evaluated
Treatment definition and timing
Outcome(s): primary and secondary, measurement frequency
Identification strategy / design selected and justification
Key assumptions and potential threats
Data sources, sample frame, pre/post periods
Primary model, covariates, bandwidths, and pre‑analysis plan
Planned robustness checks and sensitivity analyses
Decision threshold: what effect size changes the decision?