Prompt Design Recipes & Test Cases

Welcome — what this playbook helps you do

This playbook gives concise, repeatable prompt patterns and measurable test cases you can copy, adapt, and version. Use it to reduce trial-and-error, make assistant behavior predictable, and embed AI safely into everyday workflows. Each recipe includes a template, an example input and expected output, test cases with pass/fail criteria, common failure modes, and mitigations.

How to use the recipes

• Start by picking the pattern that matches your task (summarization, classification, extraction, etc.).
• Insert domain-specific examples into the template and run a small golden-test suite.
• Define pass/fail criteria before fine-tuning or production rollout.
• Automate tests where possible and store results (prompt version, test inputs, outputs, pass/fail) to build organizational memory.

Core prompt patterns (recipes + tests)

1. Summarization (concise, structured)

Template: "Summarize the following content in N sentences, preserving key facts and labels. Output as bullets with headings: Situation, Key Facts, Recommended Action."

Example input: a 400-word meeting note or report excerpt.

Expected output: 3–5 bullets under those headings that capture main facts and at least one actionable recommendation.

Test cases & pass/fail:

1. Golden case: Known document with expected 4 bullet summary — pass when summary contains all required facts and the recommended action matches expected intent.
2. Robustness: Slightly rephrased input — pass when the same facts are present (semantic match > 80%).
3. Hallucination check: Input with ambiguous data — pass when assistant marks uncertainty (e.g., "unclear: ").

Common failures: Missing facts, invented details, verbosity. Mitigation: add explicit constraints ("Do not invent facts; if missing, mark as unknown").

2. Classification (labeling with confidence)

Template: "Given the text below, assign one of these labels: [A, B, C]. Provide a one-sentence justification and a confidence score 0–100%."

Example input: short product description.

Pass/fail:

• Accuracy on golden set > target (e.g., 90%).
• Confidence calibration: flagged low-confidence results (<60%) should be routed for human review.

Fail modes: Overconfident wrong labels. Mitigation: require justification and use calibration tests to detect miscalibration.

3. Extraction / Structured data

Template: "Extract the following fields from the text: {name, date, amount, location}. Return JSON strictly with keys exactly as listed; use null for missing fields."

Example input: invoice text or meeting minutes.

Pass/fail:

1. Schema conformance (JSON valid, all keys present) — required.
2. Field accuracy on golden set > target (e.g., 95% exact match for well-formatted inputs).

Mitigation: add examples showing expected formatting and add explicit error-handling language ("If a field is ambiguous, set it to null and add a note").

4. Step-by-step / chain-of-thought for procedural tasks

Template: "You are an expert in [domain]. Show a step-by-step procedure to accomplish X, with safety checks and expected time estimates. Number each step."

Test: compare generated steps to a subject-matter expert checklist. Pass when all critical safety steps are present and ordering is correct.

Failure modes: Missing safety steps or unsafe instructions. Mitigation: include a safety filter and require citations or references for risky steps.

5. Assistant persona & role anchoring

Template: "You are with these constraints: [tone, knowledge cutoff, output format, refusal policy]. When asked X, respond in the following format: [header, analysis, recommendation]."

Test: prompt with edge-case user requests and confirm the assistant follows refusal policy and format. Pass when format is exact and refusal appears where policy dictates.

6. Conversational memory and context stitching

Pattern: Keep a compact memory summary of the session in a separate variable. Before each response, add: "Context summary: " and then the user message. Use memory for personalization only with explicit consent.

Test: simulate a 6-turn conversation and verify the assistant consistently leverages memory items and does not reveal private tokens. Pass when personalization persists where expected and privacy rules hold.

Designing evaluation test sets

Build three complementary test sets:

• Golden set: canonical examples with known correct outputs.
• Adversarial set: tricky or misleading inputs designed to trigger hallucinations, edge cases, or format breaks.
• Real-world set: sampled production inputs to measure drift and distribution changes.

Metrics to track:

• Accuracy / exact-match
• Semantic recall (for summarization/extraction)
• Hallucination rate (claims not supported by input)
• Format compliance (JSON/schema passes)
• Latency and cost per call (operational metrics)

Test automation & repeatability

Record each prompt version, model version, temperature, and cached examples. Automate tests and store results so you can compare prompt iterations over time. Use unique markers in test inputs to avoid accidental leakage between training and evaluation.

Deployment checklist & versioning

1. Define acceptance criteria (metrics and thresholds).
2. Run golden and adversarial suites. Document failures and adjustments.
3. Lock prompt and metadata into a prompt registry with semantic tags and owner.
4. Set monitoring: sample outputs daily, log user escalations, and alert on metric drift.
5. Plan rollback criteria and human-in-the-loop review for low-confidence outputs.

Common mistakes and quick remedies

• Too much freedom in the prompt — add explicit format and refuse clauses.
• No evaluation plan — design pass/fail rules before tuning.
• Assuming the model 'knows' company policies — embed or link policy text in the prompt or memory.

Next steps and experimentation ideas

Run A/B tests on competing prompt variants; keep the simpler prompt if performance is similar. Consider chaining small focused prompts instead of one large complex prompt when outputs require strict structure. Track human review rate as a primary operational KPI.

If you want this playbook to become interactive: we can add an interactive prompt-registry and test-runner so teams can store prompt versions, submit test inputs via a web form, run automated suites, and save results for reporting and trend analysis.