12. Real-Time Inference Pipeline

Real-time inference requires an optimized pipeline that delivers predictions within strict latency budgets. Every millisecond counts.

Core Pipeline Components

# End-to-end inference flow (fraud detection example)
1. Client sends transaction data (REST/gRPC)
2. API Gateway: authenticate, validate, rate-limit
3. Feature fetch: lookup user features from online feature store (Redis)
4. Feature computation: compute real-time features (tx amount vs avg, location match)
5. Cache check: has this exact transaction been scored recently?
6. Model inference: run ensemble of 3 XGBoost models (50ms budget)
7. Post-processing: apply business rules (amount > 10K → force review)
8. Response: return fraud_score + explanation + action
9. Async: log to Kafka for monitoring, feedback collection

Micro-Batching

Micro-batching groups individual inference requests into small batches to improve GPU utilization and throughput, at the cost of some latency.

# Micro-batching server logic
Incoming requests go into a queue
Every 10ms (or when queue reaches 32 requests):
  Pop all requests from queue
  Stack them into a batch tensor
  Run model inference on the batch (GPU)
  Map results back to individual responses
  Return responses

Latency added: avg 5ms (half of 10ms window)
Throughput increase: 3-5x for small LLMs

Fraud Detection Example

A complete real-time fraud detection system illustrates all the patterns:

• Input: Transaction (amount, merchant, location, device, user_id, timestamp)
• Latency requirement: p99 < 200ms (the bank needs to approve/decline before the customer leaves the checkout page)
• Scale: 5,000 TPS peak

Performance Optimization Checklist

• [ ] Profile and budget latency per component
• [ ] Use gRPC instead of REST for inter-service calls (protobuf, streaming)
• [ ] Cache features, predictions, and model weights
• [ ] Use connection pooling for DB/cache/queue connections
• [ ] Micro-batch GPU inference requests
• [ ] Pre-compute expensive features (batch → feature store)
• [ ] Quantize models (FP16/INT8) if accuracy permits
• [ ] Use async logging to avoid blocking the response path
• [ ] Warm up models (pre-load, pre-run dummy inference)
• [ ] Auto-scale based on queue depth, not CPU