13. Monitoring & Drift Detection

Infrastructure Monitoring

Metric	Alert Threshold	Action
p99 latency > 500ms	5 minutes	Auto-scale, investigate bottleneck
Error rate > 1%	2 minutes	Rollback model, page on-call
Throughput drop > 50%	1 minute	Check upstream, health of replicas
GPU utilization > 95%	10 minutes	Add GPU instances, check batching

Model Performance Monitoring

When ground truth is available (with delay), compute model accuracy:

# Model accuracy monitoring (delayed labels)
Prediction at t=0: fraud_score=0.92
Ground truth at t=24h: confirmed_fraud=True

Weekly report:
  Precision@threshold=0.5: 0.83  (was 0.85 last week) ⚠
  Recall@threshold=0.5:    0.76  (was 0.78 last week)  ⚠
  Alert if drop > 0.02

Drift Detection

Drift is the most common cause of silent ML system degradation.

Types of Drift

Type	What Changes	Detection Method
Data (feature) drift	Input features distribution shifts	KS test, PSI, chi-squared
Concept drift	Relationship between features and target changes	Accuracy monitoring, residual analysis
Label drift	Ground truth distribution shifts	Label distribution tracking
Upstream drift	Source data changes (schema, quality)	Data validation, schema checks

# Drift detection with PSI (Population Stability Index)
psi = sum((p_i - q_i) * log(p_i / q_i))
# p_i = proportion in serving data, q_i = proportion in training data
# PSI < 0.1: no drift
# PSI 0.1—0.25: moderate drift (investigate)
# PSI > 0.25: severe drift (retrain immediately)

Prometheus + Grafana Dashboard

Recommended panels:

Latency heatmap (p50/p95/p99 over time)
Throughput (requests/sec by model version)
Error rate (% by error type)
Cache hit ratio (feature + prediction)
Fallback usage rate (% of requests using fallback model)
Feature drift scores per feature (top-10 most drifted)
Model accuracy (when ground truth available)
GPU/CPU/memory utilization per replica

Alerting Strategy

P1 (immediate): System down, error rate > 5%, p99 > 2s
P2 (investigate within 1 hour): Drift detected, accuracy drop > 2%, fallback usage > 10%
P3 (next working day): Gradual performance decline, cache hit ratio drop

Exercise: Design the monitoring system for an ML-based credit scoring API. Define: a) 6 Grafana dashboard panels with chart type and metric source, b) 4 alert rules with thresholds and severity levels, c) drift detection configuration (which features to monitor, test frequency, retraining trigger), d) how you handle delayed ground truth (loans default after 90 days).