Statistical analysis & diagnostics playbook

Purpose

This playbook helps researchers choose defensible statistical approaches, run essential diagnostics, manage common data problems, control error rates, and report analyses so conclusions are robust, reproducible, and actionable. It assumes a working familiarity with common statistical methods; where helpful it points to tools and pragmatic defaults for small teams and applied researchers.

At-a-glance workflow

1. Clarify the question and estimand (what exactly you want to estimate or test).
2. Create a brief Analysis Plan (pre-specified vs exploratory).
3. Pick candidate methods/models consistent with data type and estimand.
4. Run diagnostics and validate assumptions.
5. Address missing data and influential observations.
6. Control multiple comparisons and report error metrics.
7. Run sensitivity and robustness checks.
8. Prepare reproducible reporting artifacts (data, code, environment).

1. Start with the estimand, not the test

Translate the research hunger into a clear estimand: difference in means, relative risk, hazard ratio, correlation, classification accuracy, predictive performance, etc. Document the target population, time horizon, and decision threshold. This prevents hunting for significance and anchors model choice and diagnostics.

2. Analysis planning — pre-specified vs exploratory

• Pre-specified analyses: Define primary outcome(s), primary model(s), covariates, handling of missing data, and planned subgroup analyses. Pre-specification reduces bias and simplifies multiplicity handling.
• Exploratory analyses: Label them clearly. Use them to generate hypotheses, not to make definitive claims without follow-up.

Even a short analysis plan (one page) pays off. Include version, author, date, and rationale for major choices.

3. Choosing methods matched to data and estimand

• Continuous outcome: consider linear regression (check residuals), robust regression, or transformation if indicated.
• Binary outcome: logistic regression, risk difference models, or exact tests for small samples.
• Time-to-event: Cox proportional hazards (test proportionality), accelerated failure time models when appropriate.
• Counts: Poisson or negative binomial depending on dispersion.
• Prediction tasks: separate training/validation process, cross-validation, and calibration assessment.

4. Power, sample size, and precision

Decide whether the goal is hypothesis testing or estimating with desired precision. For hypothesis testing, perform power/sample-size calculations using realistic effect sizes and variance estimates. For estimation, choose sample size so confidence intervals are narrow enough to inform decisions.

When post hoc power is considered, prefer reporting the observed effect size and confidence interval rather than post hoc power statements.

5. Handling missing data

Identify missingness mechanism (MCAR, MAR, MNAR). Practical steps:

1. Report extent and pattern of missingness.
2. Prefer principled methods over ad-hoc deletion when missingness is non-trivial:

• Complete-case analysis: acceptable when missingness is minimal and plausibly MCAR.
• Single imputation: generally discouraged for inference (biases variance).
• Multiple imputation (MI): recommended for MAR scenarios; ensure imputation model includes predictors of missingness.
• Sensitivity analysis to MNAR assumptions when plausible.

6. Multiple comparisons and error control

Decide in advance which family of tests should be controlled. Options and guidance:

• Familywise error rate (FWER): Bonferroni or Holm — conservative but simple.
• False discovery rate (FDR): Benjamini–Hochberg — useful when screening many hypotheses (e.g., -omics, exploratory outcomes).
• Hierarchical or gatekeeping procedures: pre-specify primary/secondary outcomes to reduce multiplicity burden.
• Report effect sizes and confidence intervals alongside p-values — emphasis on practical relevance.

7. Model diagnostics & goodness-of-fit

Always validate assumptions before trusting estimates.

• Linear models: residual plots (homoscedasticity, normality), leverage and influence (Cook's distance), multicollinearity (VIF).
• Logistic regression: calibration plots, ROC/AUC, Hosmer–Lemeshow (with caution), influential observations.
• Survival models: Schoenfeld residuals to test proportional hazards, martingale residuals for nonlinearity.
• Predictive models: cross-validation, holdout validation, calibration curves, and decision-curve analysis where decisions matter.

8. Sensitivity and robustness checks

Run a small battery of checks to test how conclusions change with reasonable modeling choices:

• Alternative functional forms (e.g., linear vs spline).
• Different covariate sets (minimal adjustment vs fuller adjustment).
• Different missing-data approaches (complete-case vs MI).
• Exclude influential observations or outliers and compare results.
• Placebo or negative-control analyses when possible.

9. Reporting & reproducibility checklist

Provide enough detail so competent peers can reproduce results.

• Share analysis plan, data (or a reproducible simulated dataset if data cannot be shared), and code.
• List software, package versions, random seeds, and environment info.
• Report sample sizes, missing-data counts, effect estimates with CIs, exact p-values, model diagnostics, and sensitivity analyses.
• Label exploratory findings clearly and avoid definitive language for unconfirmed discoveries.

10. Common pitfalls and red flags

• Failure to pre-specify primary outcomes or analyses (increases false discovery risk).
• Selective reporting of significant outcomes only.
• Overreliance on p < 0.05 without effect-size context.
• Ignoring missing-data mechanisms and running only complete-case analyses when missingness is substantial.
• Lack of validation for predictive models (no holdout or cross-validation).

11. Practical tools and references

• R: base stats, lm, glm, survival, mice (multiple imputation), car (VIF), broom (tidy reporting), caret and tidymodels for prediction.
• Python: statsmodels, scikit-learn, lifelines, fancyimpute / sklearn.impute for imputation.
• Guidance: ASA statement on p-values, CONSORT and STROBE reporting guidelines, textbooks on causal inference and missing data (e.g., Hernán & Robins, Little & Rubin).

Appendix A — Minimal Analysis Plan template (to copy)

Title; date; authors; version. Objective and estimand. Primary/secondary outcomes. Primary model(s) with covariates. Handling of missing data. Multiple-comparison strategy. Sensitivity analyses. Software and versions.

Appendix B — Quick diagnostic checklist

1. Is the estimand clearly stated?
2. Is the analysis pre-specified or exploratory (and labeled)?
3. Are sample-size/power or precision targets documented?
4. Is missing-data extent and treatment described?
5. Were model assumptions checked and documented (attach plots)?
6. Are multiplicity controls or families described?
7. Are sensitivity analyses performed and reported?
8. Is code and environment information included for reproducibility?

How this playbook helps

Following this playbook reduces the risk of false discoveries, improves clarity about what is being estimated, and creates a reproducible trail other researchers can audit or extend. Use it as a living document: adapt the templates, add domain-specific checks, and keep the analysis plan with the dataset and code in your project repository.