8. Serving Architecture

The serving architecture delivers predictions to consumers with low latency and high availability.

Components

Inference pipeline showing API gateway, feature fetch, model server, cache, fallback, and monitoring

Component	Role	Technology Options
Load Balancer	Distribute requests across replicas	Nginx, HAProxy, AWS ALB
API Gateway	Auth, rate-limiting, request validation	Kong, Envoy, AWS API Gateway
Inference Server	Run model, handle batching	Triton, TorchServe, vLLM, TGI
Feature Cache	Low-latency feature lookup	Redis, Memcached
Prediction Cache	Cache identical requests	Redis, custom in-memory
Model Loader	Load/unload model versions	Model registry API + local disk
Logging	Async log predictions + features	Kafka, S3, ELK

Request Flow

# End-to-end serving flow
1. Client → Load Balancer → API Gateway
2. Gateway validates auth token and rate limit
3. Gateway routes to inference service
4. Inference service:
   a. Parse request, extract features
   b. Check prediction cache (hash of features)
      - Hit → return cached prediction (skip steps c-e)
   c. Fetch missing features from online feature store (Redis)
   d. Run model inference (GPU/CPU)
   e. Log prediction + features to Kafka (async)
5. Return response to client

Inference Server Types

GPU inference: NVIDIA Triton supports dynamic batching, model ensembles, and multiple frameworks (TensorRT, ONNX, PyTorch, TensorFlow).
LLM inference: vLLM uses PagedAttention for efficient KV-cache management, supports continuous batching for high throughput.
CPU inference: ONNX Runtime optimized for CPU, good for smaller models or when GPU is not available.

Dynamic Batching

Groups multiple inference requests into a single batch to maximize GPU utilization.

# Dynamic batching
Config: max_batch_size=32, max_delay=10ms
Request A arrives at t=0:  queue (1 request)
Request B arrives at t=2:  queue (2 requests)
Request C arrives at t=5:  queue (3 requests)
At t=10: batch of 3 sent to GPU
GPU processes batch (same cost as processing 1 request, ~3x throughput)

Exercise: You are serving a text classification model (BERT-based, ~440M params) at 5,000 QPS with a p99 latency SLA of 150ms. Design the serving architecture: how many replicas, what hardware per replica (GPU/CPU/RAM), what batching strategy, what caching layers, and how do you handle traffic spikes?