Analyst Assistant Agent — Playbook & Prompts

Analyst Assistant Agent — Playbook & Prompts

This playbook helps teams design, pilot, and operate an AI assistant that amplifies analyst work without replacing human judgment. It explains what the assistant should do, safe prompt patterns and examples, data access and privacy guardrails, human‑review checkpoints, integration points with notebooks and dashboards, common failure modes and monitoring approaches, and a practical onboarding and pilot plan.

Purpose & scope

Use this assistant to speed analytic tasks that are repetitive, pattern-oriented, or preparatory for human analysis. Typical capabilities include:

• Summarizing datasets, reports, or meeting notes into concise insights for review.
• Generating exploratory questions and SQL/query prompts to guide data discovery.
• Drafting visualization specifications (chart types, axes, filters) for analysts to implement.
• Producing first‑draft narrative explanations for findings that analysts then validate.

Do not use the assistant as a final decision authority or to produce unverified compliance/legal recommendations. It should augment—never replace—human analysis and judgment.

High-level roles & responsibilities

• Analyst (owner): decides when to use outputs, validates results, and accepts final artifacts.
• Team lead or data steward: configures data access, maintains prompt templates, tracks performance metrics.
• Platform/IT: enforces access controls, logging, and deployment of the agent into notebooks and dashboards.
• Reviewer or SME: performs periodic audits of outputs for bias, accuracy, and privacy compliance.

Approved prompts & examples

Use explicit roles, expected output formats, and constraints. Prefer templates that make verification easy.

1) Summary of dataset / report

System: You are an assistant that helps analysts by producing concise, verifiable summaries. Always include a clear source list and a short list of assumptions or missing data.

User: Summarize the dataset named orders_2025_q1. Return: (a) one-sentence overall insight, (b) three notable patterns with supporting metrics or row counts, (c) two quick hypotheses to explore, and (d) required next queries. Cite the fields and filters used.

2) Query generation for exploration

System: Generate SQL queries that are safe to run on a read-only warehouse. Keep results < 1,000 rows by default and avoid cross-joins.

User: Provide up to five parameterized SQL queries to explore revenue by customer cohort and channel. For each query, include the query text, expected row count estimate, and a one-line rationale.

3) Draft visualization spec

System: Propose visualization specs for an analyst to implement. Use plain names for fields and propose appropriate chart type, axes, color, and filters.

User: For the metric 'monthly active users' broken down by product tier, propose two visualization options with short descriptions and a recommended default. Include suggested axis ranges and aggregation windows.

Always include a visible "Confidence & assumptions" section in generated outputs. Encourage the assistant to produce provenance metadata: source tables, query text, row-sample examples, timestamps, and model/version id.

Data access rules & privacy guardrails

• Follow least-privilege: the agent gets only the views/columns needed for the task. Never provide raw PII to the agent unless explicitly authorized and audited.
• Redact or tokenise sensitive fields (PII, PHI, financial identifiers) before sending any records to the agent. Prefer summary statistics over row-level records for opaque models.
• Log all queries and assistant outputs for auditability: who invoked, input prompt, returned output, data sources referenced, and timestamp.
• Apply data retention and deletion policies to logs consistent with organizational rules—avoid storing sensitive data in assistant logs where possible.
• Require explicit consent and compliance sign-off before enabling the assistant on regulated datasets.

Human review checkpoints

Design review points so a human validates all outputs used for decision-making.

1. Lightweight outputs (draft visual spec, query suggestions): analyst peer review before implementation.
2. Actionable outputs (recommendations affecting operations): sign-off by data owner and SME plus sample-based verification of results.
3. Model drift / periodic audit: monthly sampling of outputs to measure accuracy and detect bias or systematic errors.

Integration points with notebooks, pipelines & dashboards

Embed the assistant where it speeds work without breaking provenance:

• Notebook integration: provide the assistant as a sidebar or code cell tool that generates queries or prose the analyst can paste and run. Always include generated SQL and estimated row counts.
• Dashboard workflow: the assistant can suggest new tiles or filters. Treat suggestions as drafts—require an analyst to implement and test before publishing.
• Pipeline automation: use the assistant for human-reviewed template generation (e.g., alert text, investigation checklist) rather than autonomous pipeline changes.

Failure modes & monitoring

Common failure modes and mitigations:

• Hallucination: assistant invents sources or figures. Mitigation: require explicit provenance fields and automated checks comparing reported numbers to source queries.
• Data leakage: assistant reveals sensitive data. Mitigation: enforce redaction/tokenization and monitor logs for suspicious strings (SSNs, emails).
• Stale context: assistant uses outdated schema or metrics. Mitigation: include schema-version metadata and invalidate cached context on schema changes.
• Overconfidence: assistant asserts high confidence without evidence. Mitigation: require a confidence estimate and list of assumptions; route low-confidence outputs to human review.

Suggested monitoring metrics to track weekly during pilot:

• Usage: number of assistant invocations by analyst.
• Validation rate: percent of outputs that required correction or were rejected by reviewers.
• Time saved: median analyst time reduction on common tasks (self-reported or measured).
• Error rate: proportion of outputs with factual mismatches vs. source data.
• Privacy incidents: any event where sensitive data was exposed.

Onboarding & training plan

Deploy incrementally with a small pilot group and clear success criteria:

1. Select a pilot team of 4–8 analysts and identify 2–3 use cases (e.g., dataset summaries, query generation, visualization drafts).
2. Configure minimum-privilege data views and prompt templates. Train the pilot team on guardrails and review checkpoints.
3. Run a 4–6 week pilot. Collect qualitative feedback and the monitoring metrics above on a weekly cadence.
4. After pilot, evaluate: time savings >= X% (team-defined), validation rate <= Y% corrections, zero privacy incidents. If acceptable, scale gradually with continued monitoring.

Playbook checklist (quick deployment)

• Define the assistant's scope and owner.
• Build least-privilege views & redact sensitive fields.
• Create 3 approved prompt templates and example outputs.
• Instrument logging for prompts, outputs, and queries.
• Establish human review rules and audit cadence.
• Run pilot, track metrics, collect feedback, iterate.

Example prompt templates (copy-and-paste)

Keep templates simple, constrained, and provenance-focused. Analysts should customize placeholders when invoking the assistant.

Summary template: "Summarize dataset {dataset_name}. Output: one-sentence insight; three supporting metrics (with exact field names and values); two testable hypotheses; the exact SQL used (or sample if too large). Always include assumptions and list of source tables/columns."

Query template: "Generate up to five parameterized SQL queries for exploring {metric} by {dimension}. For each query include: SQL text, expected max rows, and a one-line rationale. Prefer aggregates and limit raw row outputs to <1000 rows."

Viz template: "Propose two visualization options for metric {metric} over {time_window}, include chart type, aggregation window, suggested axes, and three recommended filters."

Training materials & tips for analysts

• Teach analysts to always verify a small sample of results against source data before trusting outputs.
• Keep a short internal FAQ about known model behaviors and when to escalate to SMEs.
• Encourage users to add corrections back into prompt templates (living prompt refinement).

Preserving intent and next steps

This playbook preserves the original intent—an analyst-focused assistant to speed exploration, summarization, and visualization drafting—while adding actionable guardrails, examples, a pilot plan, and monitoring guidance. Next steps for teams: choose pilot users, configure data views, load the three prompt templates, and enable logging before first use.