Apply AI to Research & Innovation — Experiment & Literature Toolkit

Welcome — what this toolkit helps you do

This toolkit helps research teams accelerate literature review, idea generation, and experiment throughput while protecting reproducibility, attribution, and scientific rigor. Use the checklist, folder layout, and provenance template as concrete, adaptable starting points. Record the exact queries, model versions, tool settings, and file locations so others (and future you) can reproduce and audit results.

How to use this toolkit

1. Declare the research question and scope (PICO-style or equivalent).
2. Run reproducible searches and save exact queries and tool versions.
3. Screen and extract evidence using the checklist below; use AI for summary but verify sources.
4. Design experiments in the folder structure provided and record all environment and data details.
5. Log every AI interaction, data transform, and model run in the provenance template.
6. Review for bias, attribution, and reproducibility before sharing or publishing results.

1) AI-assisted literature synthesis checklist

A concise checklist to make searches and syntheses reproducible and auditable.

• Define scope and question: concise research question, inclusion/exclusion criteria, date range, study types.
• Search plan: list databases (e.g., PubMed, Web of Science, arXiv), exact query strings, filters, and search dates. Save the raw queries and results export files.
• Query engineering log: for any AI-assisted expansion of terms, save the prompt used to expand terms and the output suggestions.
• Screening protocol: screening criteria, number of reviewers, conflict resolution method. Save recruiter/annotator IDs and decisions.
• Extraction template: variables to extract (population, intervention, comparator, outcomes, methods, sample size, effect sizes, limitations).
• AI summarization rules: require the model to return direct citations, quote extracts with page/section references where possible, and a list of primary sources used. Do not accept summaries without source links or identifiers.
• Evidence matrix: a table with each included study, key metrics, quality ratings, and direct quotes supporting the extraction.
• Quality appraisal: record the tool/method used (e.g., Cochrane risk-of-bias), reviewer names, and justification notes.
• Provenance and attribution check: for each AI-derived claim, list the model name/version, prompt, and the primary sources the model cited. Verify all cited sources exist and are correctly represented.
• Search reproducibility log: save timestamps, exported result files (e.g., RIS, CSV), hash/checksum for each file, and where it is stored.

Example search log entry

{
  "date": "2026-01-23T14:12:00Z",
  "database": "PubMed",
  "query": "(type 2 diabetes) AND (continuous glucose monitoring) AND (randomized)",
  "filters": "2016-2026; human; English",
  "results_exported_to": "/project/data/search-exports/pubmed-20260123.ris",
  "export_checksum": "sha256:abcd...",
  "search_agent": "human:ajones; ai-assist:gpt-4o-research-v1",
  "notes": "AI suggested synonyms: 'CGM', 'continuous glucose monitoring devices' — accepted and added to queries"
}

2) Reproducible experiment folder structure (recommended starter layout)

Use a clear, versioned layout and include READMEs to make experiments easy to reproduce and audit. Store checksums and environment definitions with results.

/project-name/
  /docs/
    protocol.md           # study protocol, inclusion/exclusion, methods
    changelog.md
    README.html            # overview and quick start
  /data/
    /raw/                 # original immutable data (read-only)
    /processed/           # cleaned or transformed data
  /scripts/               # reusable scripts and pipeline definitions
  /notebooks/             # exploratory notebooks (annotate with versions)
  /models/                # trained models, weights, metadata
  /results/               # final figures, tables, analyses
  /provenance/            # machine- and human-readable provenance logs (json/csv)
  /env/                   # environment files: requirements.txt, environment.yml, Dockerfile
  /checksums/             # checksums for data and artifacts

Recommendations for the structure:

• Keep raw data immutable and separate from processed data.
• Include a protocol.md that describes planned analyses and pre-registration identifiers if any.
• Capture environment reproducibility: a Dockerfile or environment.yml plus exact package versions and hardware notes (GPU/CPU, memory).
• Record where sensitive data are stored and access controls; do not store credentials or PHI in plain text.
• Assign a persistent identifier (DOI or internal dataset ID) to datasets and model snapshots when appropriate.

3) Data & model provenance logging template

Use this template to capture the minimum provenance fields for each important action (search, transform, model run, human decision). Store logs as JSON or CSV in /provenance/ and keep checksums.

• timestamp — ISO 8601 when the action occurred.
• actor — human ID or agent ID (include role and contact where needed).
• action — e.g., search, extract, transform, model-train, summarize, publish.
• input_sources — list of files, dataset IDs, queries, and direct links or identifiers.
• tool_or_model — name, provider, model version, any fine-tuning details.
• prompt_or_parameters — exact prompt text or model parameters/seed used.
• output_location — file path or artifact ID where output was saved.
• output_checksum — hash of the saved artifact.
• purpose — why this action was performed.
• human_review — reviewer ID, date, and short decision note.
• license_and_attribution — source license statements or citation text for primary materials used.
• notes — free text for context, uncertainties, or next steps.

Provenance example (JSON)

{
  "timestamp": "2026-01-23T15:05:12Z",
  "actor": "ai-agent:lit-synth-v1",
  "action": "summarize",
  "input_sources": ["/project/data/search-exports/pubmed-20260123.ris"],
  "tool_or_model": "gpt-4o-research-v1",
  "prompt_or_parameters": "Summarize each study and return direct citations with PMID and quoted text where possible. Limit to 200 words per study.",
  "output_location": "/project/results/lit-summary-20260123.json",
  "output_checksum": "sha256:1234...",
  "purpose": "Rapid evidence extraction for meta-analysis draft",
  "human_review": "ajones 2026-01-24: verified citations for 18/20 entries; flagged 2 ambiguous quotes",
  "license_and_attribution": "Citations to original journals; excerpts used under fair use for research",
  "notes": "AI mis-attributed one quote—human corrected and recorded change in provenance log."
}

Prompts and practical examples

Sample prompt for literature extraction (adapt to your model):

"For each paper in the following list, produce a JSON object with keys: title, authors, year, pmid_or_doi, short_summary (<=150 words), exact_quote_with_location (if available), risk_of_bias_rating (low/medium/high) with 1-sentence justification. Do not hallucinate identifiers—if you cannot find DOI/PMID, leave field null and note 'not found'."

Sample prompt for hypothesis generation:

"Given these extracted findings (provide JSON), suggest 3 testable hypotheses that address gaps in outcomes, and for each propose a minimal experimental design, required sample size estimate (brief rationale), and primary outcome measure."

Rigor, attribution, and bias — practical checks

• Always verify AI-cited sources before including them in a manuscript.
• Declare AI use in methods, including model name and version and how outputs were verified.
• Maintain an independent human review step for all AI-synthesized extractions and interpretations.
• Run sensitivity analyses to test whether AI-derived summaries change with prompt wording or model version.
• Capture and report limitations: data availability, known model biases, and decision rules for excluding AI suggestions.

Quick-start checklist (one-page)

1. Write a 1-paragraph protocol and save to /docs/protocol.md.
2. Run reproducible searches; save exports to /data/search-exports/ and log queries.
3. Use AI to suggest synonyms and screening priorities but keep a human screening step.
4. Store extracted tables and evidence matrix in /results/ and provenance in /provenance/.
5. Capture environment (Dockerfile or environment.yml) and commit to the project repo with tags for major runs.

Where interactivity or platform capabilities can help

This toolkit works well as static guidance, but the experience can be materially improved by:

• Turning the provenance template into a saved interactive form so each action creates a recorded JSON entry (improves auditability and team practice).
• Providing a searchable evidence matrix and dataset registry backed by the provenance logs.
• Offering a packaged, adoptable toolkit that teams can copy and tailor to their governance and compliance needs.

Final notes

This toolkit is a practical starting point. Tailor the checklist, folder layout, and provenance fields to fit your domain, data sensitivity, and regulatory needs. The guiding principle: accelerate discovery while preserving an auditable chain of evidence and a clear record of human decisions.