14. Deployment Strategies

Deploying a new model version into production carries risk. The new model may have bugs, worse accuracy, higher latency, or unexpected behavior. Deployment strategies mitigate these risks by controlling how traffic moves to the new version.

Canary Deployment

Route a small percentage of traffic to the new model version (the "canary"). Monitor metrics. Gradually increase traffic if metrics are healthy.

# Canary rollout stages
Stage 1: 5% traffic → new model, 95% → old model
  Monitor: latency, error rate, prediction distribution, business metrics
  Duration: 30 minutes or until confident

Stage 2: 20% traffic → new model
  Monitor same + evaluate model-specific metrics (accuracy if ground truth available)
  Duration: 2 hours

Stage 3: 100% traffic → new model (if all metrics pass)
  Old model retained for rollback (keep 1 replica warm)

Shadow Deployment

Run the new model in parallel with the current model, but discard its responses (or log them for analysis). Users see only the current model's output. Zero user impact.

Uses: Validate latency and reliability under production load. Compare prediction distributions. Test inference infrastructure changes.

# Shadow deployment architecture
User → API Gateway → Current Model (responds to user)
                         → Shadow Model (response logged, not served)
Compare: response latency, prediction values, resource usage

A/B Testing

Route users to different model versions and measure business impact. Unlike canary (which focuses on system metrics), A/B tests focus on user-facing outcomes.

Strategy	Traffic Split	What You Measure	Duration
Canary	By request %	Latency, errors, drift	Hours
Shadow	100% to current + shadow	Latency, prediction diff	Days
A/B Test	By user cohort	CTR, conversion, revenue	Weeks

Model Rollback

Always plan for rollback. Key practices:

Keep the previous N model versions deployed (or ready to deploy).
Automated rollback triggers: error rate spikes, latency breach, drift threshold exceeded.
Feature flags for model version routing — change instantly without redeploying.
Store model artifacts immutably (no overwrites).

# Automated rollback configuration
rollback:
  triggers:
    - error_rate > 1% for 2 consecutive minutes
    - p99_latency > 500ms for 1 minute
    - prediction_drift > 0.3 compared to champion
  action:
    - route 100% traffic to previous model version
    - alert on-call engineer via PagerDuty
    - log incident with metrics snapshot

Deployment Infrastructure Patterns

Blue-Green: Two identical environments (blue = current, green = new). Switch traffic from blue to green instantly. Fast rollback by switching back.
Rolling update: Replace instances one by one. Minimal capacity drop but no instantaneous cutover.
Feature flags: Route specific users/queries to new models based on flags. Fine-grained control.

Exercise: You are deploying a new ranking model for an e-commerce search. The new model promises 10% higher conversion rate but is 2x slower. Design a deployment plan that: a) validates the latency impact on real users, b) measures the conversion rate improvement, c) allows instant rollback, d) compares the old and new model's predictions for the same query to detect anomalous changes.