Playbook

Rapid Experiment Playbook — Template & Decision Rules

An interactive short-cycle experiment template you can fill, save, and revisit: hypothesis, measurement checklist, sample-size and confidence choices, guardrails and abort rules, rollout plan, explicit decision thresholds, and a post-mortem rubric that leads to clear next steps.

Interactive Tool

Rapid Experiment Plan

Use this playbook to design short-cycle experiments that produce clear learning and practical decisions. Fill the fields below before you run the test. The form captures hypothesis, measurement and sample guidance, guardrails, abort rules, rollout plan, analysis plan, concrete decision criteria, and a post-mortem section for learnings and next steps.

Quick measurement checklist

  • Define one primary metric and its baseline.
  • Choose a realistic minimum detectable effect (MDE) and confidence level.
  • Estimate sample size or duration; prefer conservative exposure if uncertain.
  • Record guardrails and explicit abort rules up front.

Example stop/go rules

  • Abort if primary metric drops by more than X% for Y days.
  • Pause if a critical quality or safety threshold is crossed.
  • Proceed to scale only when the effect exceeds the pre-declared MDE at the chosen confidence level.

When available, attach a link to your analysis dashboard and use the post-mortem template to convert findings into actions.

Short, descriptive name (e.g., 'Checkout button color A/B').
Person accountable for running and closing the experiment.
YYYY-MM-DD or approximate. Update when you actually start.
State in plain language: 'If we [change], then [metric] will [direction] because [reason]'.
The single metric you will use to decide success (be concrete).
Current value of the primary metric (unit: % / count / rate, etc.).
Smallest relative change worth detecting (e.g., 5 for 5%).
Choose the confidence level you want for the test.
If you don't have a calculator, enter your planned sample or daily traffic and duration.
How many days you plan to run the experiment.
Metrics or events that must not be violated (e.g., error rate, safety incidents).
Write concrete triggers (e.g., 'Abort if metric X drops > 10% for 2 consecutive days').
Which users, locations, or segments will be included and how exposure ramps over time.
How you'll analyze the effect, control for confounders, and which subgroups you'll inspect.
Select outcomes you may choose based on results.
Map numeric or qualitative results to the decisions above (e.g., 'If metric >= +4% at 90% confidence => scale').
Summarize outcomes, root causes, surprises, and what the decision triggered.
Who will do what, and by when (concrete actions).
URL to the detailed results, raw data, or dashboard.
Quick assessment of potential operational or reputational risk.
Select yes when the plan is reviewed and approved.
You can explore this tool now. Sign in or create an account to save your responses and return to them later.
Sign In to Save Create an Account
Make this tool part of your work

Save a personal copy, bring it to your team, or tailor the questions and workflow to fit what you are hungry to improve.

Member customization and team collaboration are coming soon.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.