Experiment Design & Analysis Template (A/B / Causal Studies)

Purpose

This template helps teams design experiments that produce credible causal evidence and clear decisions. Use it to turn a business question into a defensible experiment: specify the hypothesis, measure what matters, ensure sufficient power, plan assignment and rollout, and follow a prespecified analysis that supports operational decisions.

When to use this template

• Testing product changes (features, UI, pricing, messaging)
• Evaluating operational interventions (process changes, training, tooling)
• Validating policy, incentive, or workflow changes that affect outcomes

Template Sections and Prompts

Problem statement & measurable hypothesis

Write a one-paragraph problem statement that explains the current situation, why it matters, and the intended change.

Hypothesis (structured):

• Action/Intervention: (what you will change)
• Target population: (who is exposed)
• Expected directional effect: (increase/decrease/no change)
• Primary outcome and expected minimum detectable effect (MDE)

Example: "Showing personalized product recommendations (intervention) to new users (target) will increase 7-day purchase rate (primary outcome) by at least 1.5 percentage points (MDE)."

Primary and guardrail metrics (definitions & tolerances)

Define each metric precisely so it can be computed from raw data without ambiguity. For each metric include:

• Metric name and short description
• Numerator and denominator (if ratio)
• Data source(s) and table/field names
• Time window for measurement
• Missing value rules
• Guardrail thresholds (what would be unacceptable) and tolerance bands

Guardrails are metrics that must not be harmed (e.g., retention, error rate, cost). Specify tolerances for each guardrail that would trigger stopping or rollback.

Measurement plan & instrumentation checklist

1. List all events/attributes to be recorded and the exact event names/fields.
2. Describe how outcomes are computed from raw events (pseudocode or SQL).
3. Confirm tests for instrumentation: event fire-rate sanity checks, duplicate IDs, and missing data thresholds.
4. Plan for data extraction timing (e.g., time-to-observe window after assignment).

Sample size & power calculation guidance

Specify the parameters used for any power calculation:

• Baseline rate or mean (from historical data)
• Minimum detectable effect (MDE) you care about
• Significance level (alpha), commonly 0.05
• Power (1 - beta), commonly 0.8 or 0.9
• Allocation ratio between arms

Notes on practical calculation:

• For proportions and two-arm tests use standard sample-size formulas or a calculator. A common approximate formula for two proportions is: estimate pooled variance at expected rates, set Z-scores for alpha and power, and solve for sample size per arm. When in doubt, use an online calculator or the platform's sample-size tool.
• Account for clustering, correlated observations, or repeated measurements by inflating sample size (design effect).
• If the expected effect is small, verify the experiment is practically feasible—the required sample may be prohibitively large.

Record the actual sample-size calculation used, the assumptions, and the date of the data used to estimate the baseline.

Randomization & assignment plan

Explain exactly how units will be assigned to treatment and control. Include:

• Unit of randomization (user, account, cookie, session, store, etc.)
• Randomization method: hash-based, server-side RNG, client-side, or batch assignment
• Blocking or stratification variables (and why they are used)
• Handling of duplicates and identity merges
• How to detect and handle cross-unit interference or contamination

Rollout, exposure, and stopping rules

Define rollout plan and operational controls:

• Start date and planned duration
• Exposure criteria (what counts as exposed)
• Interim looks: whether and how interim analyses will be performed; adjust thresholds if doing multiple looks (alpha spending or sequential methods)
• Predefined stopping rules for effectiveness, futility, or harm (include guardrail thresholds and practical actions)
• Rollback criteria and communication plan for stakeholders

Pre-analysis plan (PAP)

List all pre-specified analyses and subgroup checks. The PAP should include:

• Primary analysis method and estimator (difference-in-means, regression adjustment, survival model, etc.)
• Covariates to include and the rationale
• Handling of outliers and winsorization rules
• Missing data strategy (imputation, complete-case, etc.)
• Adjustments for multiple comparisons and planned subgroup analyses

Analysis checklist & interpretation guidance

Walk-through checklist to complete after data collection ends:

1. Data QA: confirm sample sizes, check balance across arms on key covariates, inspect assignment logs.
2. Follow the PAP exactly for the primary analysis; document any deviations with reasons.
3. Estimate treatment effect with point estimates and 95% confidence intervals; report p-values but avoid overreliance on them.
4. Check guardrails and secondary metrics; report heterogenous treatment effects if pre-specified.
5. Run sensitivity checks (alternative windows, outlier rules, bandwidths for continuous outcomes).
6. Adjust for multiple testing where appropriate (Bonferroni, Benjamini-Hochberg, or hierarchical testing).
7. Document limitations: external validity, interference, implementation deviations, and technical issues.

Decision thresholds & operational interpretation

Translate statistical results into operational actions. For example:

• If the primary outcome shows an estimated lift >= MDE with confidence interval excluding zero and no guardrail violations → recommend rollout or further scaling.
• If effect is directionally positive but uncertain or close to MDE → consider a follow-up experiment, longer run, or targeted rollout.
• If guardrails are violated or harm observed → stop and rollback immediately and investigate causes.

Operationalization & follow-through

• Who owns the decision? (person/role and timeline)
• Action plan for scaling, monitoring, and measuring long-term impact
• Plan for embedding learnings into documentation, runbooks, and product/development backlogs

Ethics, safety, and bias checks

Consider potential harms and fairness impacts. Include:

• Risk assessment for vulnerable groups
• Privacy and consent considerations
• Plan to monitor adverse events and a mitigation path

Reporting template (summary to stakeholders)

1. Experiment name, ID, start/end dates
2. Hypothesis and MDE
3. Primary result (estimate, CI, p-value) and sample sizes
4. Guardrail outcomes and any violations
5. Conclusion and recommended action
6. Limitations and next steps

Appendix: Quick templates

Hypothesis line: "If we <action> for <population>, then <primary metric> will change by >= <MDE> within <time window>."

Measurement pseudocode example (SQL-style):

SELECT user_id,
       MAX(CASE WHEN event='purchase' AND occured_at <= assigned_at + INTERVAL '7 days' THEN 1 ELSE 0 END) AS purchased_within_7d
  FROM events
  WHERE assigned_experiment='experiment_123'
  GROUP BY user_id;

Notes & best practices

• Prespecify as much as possible. Clear pre-specification reduces ambiguity and researcher degrees of freedom.
• Guardrails protect the business and should be monitored in near real time during rollout.
• Document everything: decisions, deviations, QA findings, and final interpretation so results are reproducible and auditable.
• When sample sizes are large, even trivial effects can be statistically significant. Focus on practical significance and costs/benefits.
• Consider Bayesian or decision-theoretic analyses when sequential looks or complex cost tradeoffs matter.