Statistical Intuition & Uncertainty — Cheat Sheet

Statistical Intuition & Uncertainty — Quick Reference

This one-page cheatsheet helps you read common outputs, talk about uncertainty without jargon, and check that evidence supports practical decisions. Use it as a memory aid when you review results, explain findings to stakeholders, or design a simple analysis.

Quick glossary (plain language)

• Mean — average. Useful when values are symmetric; sensitive to outliers.
• Median — middle value. Better for skewed data.
• Variance / SD — how spread out values are. SD is in the same units as the data.
• Confidence interval (CI) — a range consistent with the data and model. A 95% CI means the method would produce intervals that contain the true value ~95% of the time over many repeats, not that there is a 95% chance the true value lies inside a single computed interval.
• p-value — how surprising the observed data are under a specific null model. A small p-value says the data are unlikely if the null model were true; it does not give the probability the null is true.
• Power — chance a test detects an effect of a given size if it exists. Low power → many real effects missed.
• Effect size — how large a difference or relationship is in practical terms (e.g., % change, mean difference, or standardized d). Focus on effect size, not just significance.
• Bias — systematic error that moves estimates away from the truth (e.g., selection bias, measurement bias).

Visual intuition (what distributions tell you)

• Normal / bell-shaped — many small influences, symmetric spread; mean and SD summarize well.
• Skewed right (long tail) — median often better; mean pulled by large values.
• Bimodal — likely two underlying groups; the overall mean can be misleading.
• Heavy tails — more extreme values than normal; CIs and tests relying on normality may be misleading.

Interpreting p-values and confidence intervals (plain language)

• p < 0.05: evidence against the null model at a conventional threshold, not proof. Ask: was the test planned? Were there many comparisons? Is the effect practically meaningful?
• CI includes zero (or the null value): data are compatible with no effect and with some non-zero effects inside the interval. Emphasize the range of plausible values.
• Small p-value but tiny effect size: statistically significant may be practically negligible. Always report both.
• Wide CI: results are imprecise. Wider intervals come from small samples or noisy measures.

Rules of thumb for sample size & power

• General principle: more variability in the outcome or smaller target effects require larger samples.
• For continuous outcomes standardized by SD=1, a quick rule: n (per group) ≈ 16 / d² gives ~80% power for a two-sample comparison, where d is Cohen's d (effect size in SD units). Example: d=0.5 → n≈64 per group; d=0.2 → n≈400 per group.
• For proportions near 50%, detecting small differences (≈5 percentage points) typically requires on the order of 1,000–5,000 observations per group; detecting 10-point differences often needs hundreds per group; large differences (≈20 points) may need only tens to a few hundred. Use this as a rough ordering, not an exact calculation.
• If in doubt, run a simple sample-size calculation (there are reliable online calculators) before collecting data. Make the minimal detectable effect an explicit planning choice tied to practical impact.

Checklist for communicating uncertainty to non-technical stakeholders

1. Start with the decision question: what decision would change if the effect exists at the lower vs. upper bound of the CI?
2. Report the effect size and a plain-language CI (e.g., “we estimate an increase of 2–6 percentage points”).
3. State the sample size and any important data quality issues (missing data, measurement noise, selection concerns).
4. Be explicit about limitations: multiple tests, unplanned analyses, confounders, or non-randomized evidence.
5. Use visuals: show the CI on the same scale as the practical threshold (e.g., revenue, defect rate) and label what counts as a meaningful change.
6. Offer alternative plausible explanations and recommend next steps (replication, experiment, stronger control, or monitoring).

Short examples — quick reads

A/B test

Result: variant B shows a 1.8% lift, p=0.03, 95% CI [0.2%, 3.4%]. Interpretation: data suggest a small positive effect; CI shows the true lift could be very small or modest. Ask whether a ~1–3% lift justifies rollout given cost and risk. Check multiple comparisons, randomization quality, and sample size.

Time-series / pre-post change

Don’t conclude causation from a single before/after measure. Check for ongoing trends, seasonality, and confounding events. Prefer interrupted time-series with enough pre- and post-observations or a control series.

Correlation vs causation

Correlation indicates association, not directionality. Ask: could a third factor explain both variables? Use experiments (randomized), natural experiments, or causal diagrams and adjusted models to edge closer to causal claims.

Common pitfalls & quick remedies

• Cherry-picking: avoid reporting only tests that ‘worked.’ Pre-specify primary outcomes when possible.
• Overinterpreting non-significant results: absence of evidence is not evidence of absence—report CIs and power considerations.
• P-hacking: be skeptical of borderline p-values from many explored analyses. Transparency about the analysis path helps trust.
• Ignoring practical significance: always ask whether the magnitude of effect matters for the real decision.

Quick language to use (and avoid)

Use: “consistent with,” “compatible with,” “estimate,” “range of plausible values,” “we have X% confidence in the method,” “this effect would likely/possibly change decision Y.”

Avoid: “this proves,” “this shows the probability the hypothesis is true is X%,” or reporting p-values without context.