Meta-analysis & systematic review playbook

Why this playbook

Need to combine evidence from multiple studies without being misled? This playbook helps you plan and run a reproducible systematic review and meta-analysis that reduces bias, avoids common pooling mistakes, and produces clear, decision-ready results.

Core steps (at-a-glance)

1. Register a protocol and frame a clear PICO (or equivalent).
2. Design and run comprehensive searches; manage and de-duplicate results.
3. Screen, select, and document study inclusion using transparent criteria.
4. Extract standardized data; assess study-level risk of bias.
5. Choose appropriate effect measures and pooling models; evaluate heterogeneity.
6. Run sensitivity, subgroup, and publication-bias analyses.
7. Report results following standards (e.g., PRISMA) and assess certainty (e.g., GRADE).

1. Protocol & question framing

Register a protocol before you start screening. Use PROSPERO, OSF, or an institutional registry. A registered protocol protects against selective reporting and post-hoc analytic choices.

Frame a precise question using PICO (Population, Intervention, Comparator, Outcome) or alternate frameworks (PECO, SPICE) and pre-specify key analyses and thresholds for inclusion, subgroup tests, and sensitivity checks.

2. Search strategy & record management

Build sensitive search strategies across multiple databases (e.g., PubMed/MEDLINE, Embase, CENTRAL, Web of Science), use controlled vocabulary + free text, and document search strings and dates.

• Include grey literature, trial registries, conference proceedings, and contact authors when appropriate to reduce publication bias.
• Export results to a reference manager or screening tool and run systematic de-duplication (record matching by title/DOI/authors/date).
• Keep a reproducible search log (queries, database, date, number of hits).

3. Screening & selection

Use two independent reviewers for title/abstract and full-text screening where possible. Resolve disagreements by consensus or a third reviewer. Record reasons for exclusion at full-text review.

Create and store a PRISMA flowchart showing records identified, screened, eligible, and included.

4. Data extraction & templates

Design an extraction template and pilot it on a sample of studies. Key fields to capture:

• Study identifiers (authors, year, DOI)
• Design and setting (RCT, cohort, diagnostic study, sample size)
• Population characteristics (age, sex, comorbidities)
• Intervention/comparator details and timing
• Outcome definitions, measurement tools, and time points
• Effect estimates (raw counts, means/SDs, adjusted estimates, CIs) and how they were derived
• Notes on missing data, cross-over/clustering, or multiple arms
• Funding sources and conflicts of interest

Record who extracted each item and keep an audit trail of changes. Consider using an interactive extraction form to store structured responses (see Capability notes).

5. Risk-of-bias assessment

Select tools appropriate to study designs:

• Randomized trials: Cochrane RoB 2
• Non-randomized studies: ROBINS-I
• Diagnostic accuracy: QUADAS-2
• Observational studies: Newcastle–Ottawa Scale or tailored RoB checklist

Assess domains (selection, performance, detection, attrition, reporting) and summarize overall risk. Use judgments to plan sensitivity analyses (e.g., exclude high risk-of-bias studies).

6. Choosing effect measures & pooling models

Decide effect measures before analysis (risk ratios, odds ratios, mean difference, standardized mean difference). Be consistent and explain conversions.

Model choices:

• Fixed-effect assumes a single true effect — rarely appropriate across diverse studies.
• Random-effects assumes underlying distribution of effects — usually more realistic; choose robust estimators (REML, or use Hartung–Knapp for CIs).

Beware of common pooling mistakes: combining incompatible outcomes, mixing adjusted with unadjusted effect estimates without caution, incorrectly handling multiple treatment arms, or treating dependent effect sizes as independent.

7. Heterogeneity & exploration

Quantify heterogeneity with I2 and τ2, and report prediction intervals where helpful. Avoid binary interpretation of I2; consider absolute heterogeneity and clinical context.

Explore heterogeneity via prespecified subgroup analyses and meta-regression where data permit. Avoid data-dredging—pre-specify hypotheses in the protocol.

8. Sensitivity analyses & robustness checks

Pre-plan sensitivity checks such as:

• Exclude high risk-of-bias studies
• Use alternative pooling estimators
• Exclude small or outlying studies
• Different handling of missing or imputed data

Report how conclusions change (or not) across these analyses.

9. Publication bias & selective reporting

Assess small-study effects and publication bias using funnel plots (when ≥10 studies), Egger's test, trim-and-fill, or selection models. Interpret all such tests cautiously—the absence of evidence is not evidence of absence.

Reduce risk by searching grey literature, trial registries, and contacting authors for unpublished data.

10. Certainty of evidence & interpretation

Use GRADE to assess certainty across outcomes, considering risk of bias, inconsistency, indirectness, imprecision, and publication bias. Translate statistical findings into practical implications for decision-makers (absolute risk change, NNT, prediction intervals).

11. Reporting & reproducibility

Follow PRISMA 2020 for transparent reporting. Include:

• Protocol registration details and deviations
• Complete search strategies and dates
• PRISMA flow diagram
• Extracted data tables and risk-of-bias summaries
• Forest plots, heterogeneity statistics, funnel plots, and sensitivity analyses
• Availability of analytic code and data (e.g., GitHub, institutional repository)

Quick practical checklist

1. Register protocol (PROSPERO/OSF) — include PICO and planned analyses.
2. Run multi-database searches and document them.
3. De-duplicate and screen with at least two reviewers.
4. Pilot and use a structured extraction template; capture effect sizes and variances.
5. Use appropriate RoB tools per design and plan sensitivity tests based on RoB.
6. Select effect measure and justify pooling model; report I2, τ2, and prediction intervals.
7. Assess publication bias and report limitations transparently.
8. Publish data, code, and a PRISMA-compliant report.

Tools & resources

• Risk-of-bias: RoB 2, ROBINS-I, QUADAS-2
• Guidance: Cochrane Handbook, PRISMA 2020, GRADE handbook
• Software: RevMan, R (metafor, meta, metaforest), Stata (metan/meta), Comprehensive Meta-Analysis
• Repositories: PROSPERO, OSF, GitHub for code and data sharing

Common pitfalls & how to avoid them

• Pooling incompatible outcomes — harmonize outcomes or report separately.
• Mixing adjusted/unadjusted estimates — prefer consistent types or run sensitivity analyses.
• Over-reliance on I2 thresholds — interpret in context and report τ2 and prediction intervals.
• Opaque selection decisions — document inclusion/exclusion reasons and search strategy fully.

Next steps & templates

Suggested immediate actions: register a protocol, draft PICO and inclusion criteria, assemble search strings, and create a shared extraction template. If you want, convert the extraction template into an interactive form to capture extractions and RoB assessments centrally.

References & further reading

Include references to Cochrane Handbook, PRISMA 2020 statement, GRADE guidance, and key meta-analysis methodology papers. (Add specific citations in your hosted version.)