16. Monitoring & Observability

AI systems require monitoring at two levels: infrastructure (is the system healthy?) and model (is the model still making good predictions?).

Infrastructure Metrics

Model Performance Metrics

• Prediction distribution: Histogram of model scores. A sudden shift may indicate data drift.
• Feature distribution: Compare each feature's distribution against training baseline (Kolmogorov-Smirnov test, Population Stability Index).
• Model accuracy: When ground truth is available (with delay), compute accuracy, precision, recall, F1.
• Drift scores: Data drift (feature distribution shift) and concept drift (relationship between features and target changes).

# Drift detection example (Evidently AI)
Feature: "transaction_amount"
Training mean: 125.4, std: 89.2
Serving mean: 178.1, std: 112.5
KS statistic: 0.23 (threshold: 0.1)
→ ALERT: transaction_amount feature has drifted significantly
→ Possible causes: new merchant category, seasonal effect, data pipeline bug

Prometheus + Grafana Stack

Standard open-source monitoring stack for AI systems:

• Exporters: Prometheus metrics exposed by inference servers (Triton, TorchServe), application code, infrastructure.
• Prometheus: Time-series database, collects and stores metrics.
• Alertmanager: Routes alerts to PagerDuty, Slack, email.
• Grafana: Dashboards with per-model, per-version, per-feature views.

Distributed Tracing

Trace requests across microservices to identify bottlenecks:

# Trace: single inference request
Span 1: API Gateway → 5ms
Span 2: Auth → 3ms
Span 3: Feature fetch → 22ms (Redis)
Span 4: Feature compute → 15ms
Span 5: Model inference → 87ms (GPU)
Span 6: Post-process → 8ms
Span 7: Log to Kafka → 2ms (async)
Total: ~142ms

Logging Strategy

• Request logs: Input, output, latency, model version, feature snapshot (for later debugging).
• Prediction logs: All predictions with metadata for offline analysis and retraining.
• Feedback logs: Ground truth (when available) for accuracy computation.
• System logs: Errors, warnings, resource usage.

# Structured log entry (JSON)
{
  "timestamp": "2026-06-15T10:30:00Z",
  "request_id": "req_abc123",
  "model_id": "fraud-detection-v3",
  "latency_ms": 142,
  "input_features": {"amount": 299.99, "merchant": "amazon", ...},
  "prediction": {"fraud_score": 0.12, "action": "approve"},
  "cache_hit": false,
  "fallback_used": false
}