KPI Definitions & Interpretation Guide

Experiment velocity

Definition: The rate at which a team launches or completes distinct experiments in a defined time window. Measures throughput of learning activity.

Formula: Experiment velocity = Number of completed experiments in period / length of period (e.g., experiments per week or per month)

Required data: Experiment registry entries with start date, completion date, experiment status, owner. Data source: experiment tracking system or team log.

Units & time window: experiments/week or experiments/month. Always state the window used.

Calculation notes & edge cases: Define what counts as a single experiment (hypothesis + test & measurable outcome). For parallel A/B tests against the same hypothesis, count either as one experiment with multiple variants or as separate experiments depending on your taxonomy — document the choice. Only count experiments that reached a prespecified completion criterion (success, failure, or sufficient learning) to avoid inflating velocity with abandoned work.

Interpretation & examples: Higher velocity indicates more learning cycles, but quality matters. Example: 8 completed experiments over 4 weeks → 2 experiments/week. If velocity rises but signal-to-noise is low, learning may be superficial.

Visualization tip: Use a cumulative experiments timeline and a rate chart (rolling 4-week average) rather than a single-period snapshot.

Governance: Owner — Experiment Lead. Versioned definition. Note minimum completion criteria.

Signal-to-noise ratio (SNR) in experiments

Definition: A practical measure comparing the magnitude of the observed effect (signal) to the variability or measurement error (noise). Helps judge whether an observed change is likely meaningful.

Practical formula: SNR ≈ |Observed effect size| / Estimated measurement error (standard deviation of the estimate)

Required data: Baseline metric, treatment metric, sample sizes, and either repeated measurements or an estimate of measurement variability (standard error). Data source: experiment analytics platform or statistical analysis outputs.

Units: unitless.

Calculation notes & edge cases: For proportion metrics, standard error = sqrt(p*(1-p)/n). For continuous metrics, use the standard error of the difference. Small samples inflate uncertainty; label SNR calculations with the underlying sample size. When measurement error is unknown, estimate conservatively or perform a power analysis before treating results as conclusive.

Interpretation & examples: SNR > 1 suggests the observed effect is larger than the typical noise; larger SNR gives more confidence. Example: observed lift = 0.02 (2 percentage points), SE = 0.005 → SNR = 4 (stronger evidence). If SNR < 1, treat result as inconclusive and collect more data or refine measurement.

Visualization tip: Show effect sizes with error bars or confidence intervals; plot SNR alongside sample size to highlight underpowered tests.

Governance: Owner — Data Science or Experimentation Lead. Document the method for estimating measurement error.

Conversion / impact per experiment

Definition: The measured change in a target conversion metric attributable to an experiment, expressed as absolute change, percent change, or per-experiment impact.

Formulas (examples):

• Absolute lift = Conversion_rate_treatment − Conversion_rate_control
• Relative lift (%) = (Conversion_rate_treatment − Conversion_rate_control) / Conversion_rate_control × 100
• Impact (units) = Absolute_lift × number_of_eligible_users (gives expected additional conversions)

Required data: Treatment and control conversion counts and denominators, sample split details, attribution window. Data source: tracking events, analytics platform.

Units & time window: Percentage points, percent, or absolute units (conversions). Always specify the attribution window (e.g., 7 days after exposure).

Calculation notes & edge cases: Use consistent denominators (eligible users exposed vs. all users). Account for test leakage, cross-contamination, and variant weighting. When using revenue or value-per-conversion, include currency and time frame. If using per-experiment impact forecasts (extrapolating lift to population), include confidence intervals and assumptions.

Interpretation & examples: Control conversion = 5%, treatment = 8% → absolute lift = 3pp, relative lift = 60%. If 10,000 eligible users, expected +300 conversions. If SNR is low or sample bias exists, label impact as provisional.

Visualization tip: Show side-by-side conversion rates with confidence intervals and a table with extrapolated impact (low/median/high scenarios).

Governance: Owner — Product/Experiment Owner. Document attribution rules and population filters.

Cost-of-delay (CoD) estimates

Definition: A monetary estimate of the value lost (or value deferred) per unit time when delivery of a feature, experiment, or initiative is delayed. Useful for prioritization (e.g., WSJF-style ranking).

Simple estimation formula (guideline):
CoD = Estimated value (expected benefit over horizon × probability of value realization) / Time period over which delay matters
For prioritization, WSJF = CoD / Job duration (or implementation effort).

Required data: Expected benefit (revenue uplift, cost savings, retention improvements converted to $), probability of success, expected useful life or horizon, and estimated implementation duration. Data sources include finance, product forecasts, and experiment results.

Units: currency per unit time (e.g., $/week).

Calculation notes & edge cases: CoD estimates are inherently uncertain—capture ranges and scenarios (low/median/high). Include non-revenue impacts (reputational risk, regulatory deadlines) qualitatively or with proxy values. For long-lived features, discount future benefits appropriately or keep a clearly stated horizon.

Interpretation & examples: If expected additional monthly revenue = $30k with 50% probability of realization, and the value is concentrated in the next 3 months, expected value = $15k over 3 months → CoD ≈ $5k/month. WSJF then divides this by estimated implementation months.

Visualization tip: Present CoD as ranges and show WSJF rank alongside effort estimates to facilitate prioritization conversations.

Governance: Owner — Product Finance or Portfolio Lead. Require documented assumptions and sensitivity scenarios.

Adoption and retention measures for pilots

Definition: Measures that describe initial uptake (adoption) and continued usage (retention) among pilot participants or early users.

Common formulas:

• Adoption rate = Number of users who used the feature at least once / Number of targeted pilot users
• Day N retention = Number of users active on day N after first exposure / Number of users active on day 0 (or initial use)
• Cohort retention curve = retention rate plotted across days/weeks for a defined user cohort
• Churn rate = 1 − retention rate over the chosen window

Required data: User identifiers, timestamped usage events, cohort definitions, sample population. Source: product analytics and user logs.

Units & time window: Percentages; define cohort and time window (Day 1 / Day 7 / Day 30, or weekly cohorts).

Calculation notes & edge cases: For pilots, sample sizes are often small — report counts alongside percentages. Watch for selection bias: pilot users may not represent the broader population. Define what qualifies as "active" (e.g., session, feature call, meaningful action).

Interpretation & examples: A pilot with adoption 40% and Day 7 retention of 12% suggests initial interest but weak continued value for most users; consider onboarding or product-market fit issues. Use cohort curves to identify where drop-off happens.

Visualization tip: Cohort retention heatmaps and retention curves are most effective. Always show absolute user counts to contextualize percentages.

Governance: Owner — Product/Analytics. Document cohort definitions and the rule for qualifying as "adopted".