Model Serving Reference Architecture (batch, online, hybrid)

Purpose and scope

This playbook helps teams choose and implement model-serving approaches that match real latency, throughput, cost, and reliability needs. It describes four common patterns—batch, online (low-latency), streaming, and hybrid—showing when each fits, core components, operational controls, common failure modes, and practical recipes for autoscaling, caching, warmstart, security, and SLO mapping.

Why this matters

Mismatched serving choices cause surprising latency spikes, runaway costs, poor customer experience, and operational toil. The right pattern minimizes risk while meeting business SLOs and operational constraints.

Quick decision guide (short)

• Use batch when predictions can be delayed (minutes–hours), cost per prediction must be low, and throughput is high.
• Use online serving when user-facing latency must be low (ms–single-digit 100s ms) and per-request freshness matters.
• Use streaming when you must continuously score events with low-to-moderate latency and maintain stateful feature computation.
• Use hybrid when you need a mix (e.g., low-latency lookup for hot keys + batch backfill for bulk scoring or model retraining).

Pattern details and tradeoffs

Batch serving

Architecture: scheduling/orchestration (cron / Airflow / K8s CronJob) -> feature retrieval -> batch scoring job -> storage (DB / object store) -> downstream consumers or dashboards.

Pros: cheapest per-prediction at scale, simpler to test and audit, easier to reproduce results. Cons: prediction latency high, not suitable for user-facing decisions requiring freshness.

Typical uses: monthly risk scoring, nightly ETL scoring, large retraining evaluation, deferred personalization.

Online (low-latency) serving

Architecture: client request -> edge / API gateway -> auth & input validation -> model inference service (microservice or managed inference) -> feature store or cache -> response. May sit behind CDN or regional gateways for geo-distribution.

Pros: low latency, immediate responsiveness. Cons: higher cost per prediction, requires careful autoscaling, warmstart handling, and state/versioning.

Typical uses: recommendation APIs, fraud decisions at checkout, conversational assistants, live personalization.

Streaming serving

Architecture: event stream (Kafka / Pulsar) -> stream processing (Flink / Spark Structured Streaming / Beam) -> stateful transformations -> inline model scoring or callout to inference serving -> sink to materialized views or alerting systems.

Pros: continuous processing, good for event-driven pipelines and stateful feature windows. Cons: more complex to test and observe, can have higher operational overhead.

Typical uses: real-time risk scoring on event streams, sensor data processing, clickstream enrichment.

Hybrid patterns

Combine patterns to get the best of both worlds. Example: use online serving for hot keys and cache hits, fall back to an asynchronous batch job for cold keys or heavy computations. Or use streaming to precompute features and an online service to make the final prediction.

Hybrid benefits: cost-efficient low-latency for common cases, graceful degradation for heavy loads. Pitfalls: complexity in consistency, cache invalidation, and operational visibility.

Core components and responsibilities

• API & gateway: input validation, auth, rate limiting, request tracing.
• Model repository & versioning: immutable model artifacts, metadata, provenance.
• Feature store or feature cache: online store for low-latency lookups; offline store for batch features.
• Inference runtime: microservice, serverless function, or managed inference endpoint (GPU/CPU-based).
• Orchestration & scheduling: CI/CD, canary/blue-green deploys, retraining pipelines.
• Monitoring & observability: latency, throughput, error rates, data drift, concept drift, feature distribution changes.
• Storage & sinks: result stores, audit logs, materialized views for downstream users.
• Security & compliance: access controls, encryption, PII handling, audit trails.

Operational controls and best practices

Autoscaling

Design autoscaling based on multiple signals: request concurrency, CPU/GPU utilization, request queue length, and custom business metrics (e.g., QPS per model). Prefer conservative ramp-up rules and warm pools to avoid cold-start latency. Use a combination of horizontal pod autoscaling and node pool autoscaling where available.

Warmstart

Keep a small pool of warmed inference instances for critical low-latency paths. For serverless or cold-start-prone environments, use lightweight health probes and periodic keepalive traffic to reduce cold starts. For GPU models, consider multi-model servers or model sharing strategies to reduce load times.

Caching strategies

Cache predictions for idempotent or repeatable requests. Use short TTLs for freshness-sensitive results. For compound results, cache at multiple levels: CDN/edge for identical requests, application-level cache for recent queries, and feature caches for online feature retrieval.

Model versioning and rollout

Use explicit model version IDs and keep backward-compatible feature contracts. Deploy with canary percentages and compare live metrics against a control. Keep an automated rollback path if error or business metric regressions cross thresholds.

SLO & security mapping

Map business SLOs to technical SLOs—e.g., business target: checkout decision within 200ms for 99% of requests. Technical SLOs: 95th percentile inference latency < 150ms, cache hit rate > 70%, error rate < 0.1%.

Security mapping: identify data classification (PII, PHI), enforce encryption in transit and at rest, implement RBAC for model artifacts and telemetry, and log access for audits.

Monitoring signals and alerts

Key signals: end-to-end latency (p50/p95/p99), request throughput, error rate, model confidence distribution, feature distribution / drift, input schema violations, cache hit ratio, cold-start counts, resource utilization, and business KPIs tied to model output.

Recommended alerts: sudden shift in input distribution, rising p99 latency, elevated error rates, model-score distribution drift beyond baseline thresholds, and sustained increase in cold starts.

Checklist: choosing the right serving mode

1. What is the max acceptable end-to-end latency for the decision? (ms, seconds, minutes)
2. Is the prediction user-facing or background? (user-facing -> online; background -> batch)
3. What is expected QPS and burstiness? (low & predictable vs high & spiky)
4. How fresh must features/predictions be? (real-time, near real-time, daily)
5. Are predictions idempotent and cacheable? (yes -> caching helps)
6. Does the model require GPU or special hardware? (affects cost & cold-start strategy)
7. What are data governance and compliance constraints? (PII, retention, audit)
8. What rollback and testing capabilities do you have? (canary, shadowing, shadow traffic)
9. What observability and alerting are in place for production drift? (monitoring coverage)
10. Does the team have ops maturity to run streaming or stateful services? (if not, prefer managed/less-complex patterns)

Example quick mappings

• High QPS, low freshness requirement -> Batch (or hybrid with cached hot keys).
• Low-latency critical decision, moderate QPS -> Online with warm pools and autoscaling.
• Event-driven per-message scoring with stateful windows -> Streaming.
• Recommendation with heavy feature engineering and periodic retrain -> Hybrid (stream precompute + online join).

Next steps & templates

Use the checklist above to classify each model use-case. For teams that want practical templates, consider these follow-ups:

• Interactive serving decision worksheet (collects latency, QPS, freshness, cost constraints and recommends a pattern).
• Deployment templates: canary/blue-green configs, HPA settings, node-pool suggestions for GPU/CPU mixes.
• Observability dashboard templates: latency percentiles, feature drift, cold-starts, cache hit ratio.

Image & diagram suggestions

Suggested search phrase for illustrative diagrams: "microservice architecture model serving". Diagrams to include in your copy: request flow (online), batch ETL flow, streaming topology, hybrid fallbacks (cache + batch backfill), and SLO mapping matrix.

Closing

This playbook is a working reference: begin with the checklist, pick the simplest pattern that meets SLOs, instrument observability early, and iterate. If you want, convert the checklist into an interactive worksheet to capture requirements and produce a recommended serving pattern automatically.