Connector & Integration Troubleshooting Runbook

Purpose & Scope

This runbook helps engineers, data platform operators, analysts, and on-call responders quickly diagnose and remediate connector and integration failures. It covers: connectivity failures, authentication & permissions, schema drift and transformation mismatches, data latency and backfill procedures, logging & metrics checks, and communication & escalation templates. Use this runbook for production and pre-production connectors that move upstream operational data into analytics, data warehouses, or downstream systems.

When to use

• Dashboards show missing or stale data.
• Connector health alert fired (failed job, repeated errors, or large drop in row counts).
• New schema deployment or source system change suspected.

Prerequisites & Ownership

Keep the following up to date in the connector's documentation: primary owner, on-call rotation, service account names, environment-specific endpoints, and runbook version. Confirm required access for troubleshooting (SSH, management console, cloud project, logs, metrics, schema registry).

Runbook owner: Data Platform Team / Connector Owner

Quick Triage Checklist (first 10 minutes)

1. Confirm the impact: Which downstream reports/tables are stale or incorrect? Note affected datasets and SLAs.
2. Check connector status in orchestration system (Airflow, Prefect, Control Plane). Note last successful run and error timestamps.
3. Look at recent error messages and last stack traces in logs. Capture earliest error time and sample error text.
4. Check system-level metrics: job duration, failure rate, queue backlog, CPU/memory of the connector host, and network errors.
5. If incident is producing customer impact or large data loss, follow the escalation path (see Escalation section).

Troubleshooting Flow (high level)

1. Scope & impact → identify affected flows and datasets.
2. Connectivity & credentials → confirm network and auth.
3. Schema & transformation → detect schema drift or mapping errors.
4. Volume & latency → detect backfill needs or slow ingestion.
5. Fix or mitigate → quick fix or rollback; schedule backfill if needed.
6. Communicate & escalate → inform stakeholders and document incident.

1) Connectivity & Environment

• Verify network reachability: ping or curl the source and destination endpoints from the connector host or container. Example: curl -v https://source-api.example.com/health.
• Check DNS and routing if the host cannot resolve the endpoint.
• Confirm environment-specific variables (dev/prod endpoints) and that the connector is running in the correct environment.
• Inspect orchestration logs for transient network timeouts vs persistent failures.

2) Authentication & Permission Checklist

Common causes: expired keys, rotated service accounts, insufficient permissions, or scope changes.

1. Confirm credentials in use: service account key, API token, OAuth client ID/secret. Check secrets manager for last rotation date.
2. Test credentials manually (where safe): use the same token to call a minimal API endpoint or database health query.
3. Verify IAM permissions: read/list/select permissions on source and write/insert permissions on destination.
4. Check if credential rotation coincided with failures. If rotated, ensure the connector config was updated.
5. For OAuth flows, confirm refresh tokens are valid and token exchange logs are successful.
6. If credentials are suspected compromised or corrupted, follow security policy and rotate credentials, then redeploy connector config.

3) Schema Drift Detection & Mitigation

Schema drift can cause failed loads, silent data truncation, or incorrect downstream metrics. Use these checks:

1. Compare source and last known schema:
   • Pull current source schema (API describe, SHOW COLUMNS, schema registry).
   • Diff against the schema used by the connector/transformation stage (store a schema snapshot in repo or registry).
2. Look for changes in data types, added/removed fields, or nested structure changes.
3. Run quick row-level validation: compare record counts over a recent window and null rates for key fields.

   SELECT count(*) FROM source_table WHERE event_time > now() - interval '24 hours';
   SELECT count(*) FROM staging_table WHERE event_time > now() - interval '24 hours';
   

4. If type mismatches are found, consider temporary workarounds: cast fields as strings, add tolerant parsing, or drop problematic fields to restore flow while planning a mapping change.
5. Record schema change events and notify downstream consumers before permanent changes.

4) Transformation & Mapping Mismatches

• Validate transformation code against a known-good sample of source data.
• Re-run transformation locally with a subset of data to reproduce errors and inspect stack traces.
• Check for silently dropped records (filtering logic) and incorrect joins that reduce cardinality.

5) Data Latency, Backfill & Recovery Procedures

1. Quantify the gap: identify missing time ranges and affected partitions.
2. Decide the recovery approach:
   • If source supports replay by time or ID, request replays for the gap window.
   • If only bulk export is available, plan a controlled bulk load into staging and run transformations.
3. Backfill plan checklist:
   • Estimate rows and compute costs/time.
   • Run backfill on a staging path first; validate counts and key metrics.
   • Schedule backfill during low-traffic window if it impacts downstream systems.
   • Monitor for duplicates — ensure idempotent write patterns or run dedupe after backfill.
4. Validation post-backfill: row counts, sample records, hash or checksum of critical columns, and reconciliation with expected aggregates.

6) Logs, Metrics & What to Check

• Connector job logs (orchestration): job start/end, exit code, stack traces.
• Application logs: parser errors, serialization exceptions, null pointer, rate-limited responses.
• Source API/database metrics: 4xx/5xx rates, throttling headers, slow query logs.
• Monitoring dashboards to check: success rate, latency histograms, event counts, and error classifications.

  Use these to determine whether error is transient (rate limit) or structural (schema change).

7) Common Error Patterns & Suggested Fixes

• Authentication errors (401/403): check keys, token expiry, and IAM roles.
• Network timeouts: check retries, increase timeouts for large calls, or investigate network issues.
• Deserialization/parsing exceptions: add tolerant parsing, better schema validation, or sanitize inputs.
• Zero or low row counts: inspect filters, timezones, and partition predicates; check source retention policy.
• Duplicate records: review idempotency key logic; add deduplication step if necessary.

8) Escalation & Stakeholder Communication Templates

Use short, actionable messages. Include what happened, impact, immediate mitigation, and next steps.

Slack / Chat template

@channel: Connector [connector-name] failed at [time]. Impact: [datasets/dashboards] are stale. Triage started. Owner: [owner-name]. Next update in 30 minutes or when there is a change.

Email update template

Subject: Incident: [Connector-name] — Data Delay / Failure

Summary: Brief description of what failed and when.
Impact: Affected datasets and business impact.
Immediate mitigation: What was done (paused job, temporary fix, initiated backfill).
ETA: Expected next update and rough recovery timeline.
Owner: Contact info for the owner.

9) Post-Incident Checklist

1. Document root cause and timeline in the incident log.
2. Runbook updates: add missing checks, add helpful logs, or change alert thresholds.
3. Create preventative actions: schema contracts, versioned schemas, automated schema tests, canary runs for deploys.
4. Schedule a short review with downstream consumers to confirm data validity and adjustments needed.

Appendices

Minimal Flowchart (text)

1. Start → Is connector running? → No: restart & monitor; Yes: next
2. Are there auth errors? → Yes: check credentials; No: next
3. Are there schema errors? → Yes: run schema diff & mitigate; No: next
4. Is data delayed? → Yes: estimate gap & backfill; No: close incident

Useful Commands & Queries (examples)

# Example curl health check
curl -I https://api.example.com/health

# Quick row count compare (SQL)
SELECT count(*) FROM source_table WHERE event_time > now() - interval '24 hours';
SELECT count(*) FROM staging_table WHERE event_time > now() - interval '24 hours';

Where to store artifacts

Keep logs, screenshots, diffs, and validation queries attached to the incident ticket and in the connector's documentation folder (e.g., repo / Confluence page / shared drive).

Notes & Safety

Do not share credentials in chat or email. When rotating keys or changing IAM, follow security change procedures. This runbook is a procedural guide and must be adapted to environment-specific policies, governance, and compliance requirements.