12. Fault Tolerance

ML systems must handle failures gracefully. Unlike traditional services where failing with a 500 error may be acceptable, ML systems often serve critical decisions where a wrong answer or no answer has real consequences.

Replication

Deploy multiple copies of every critical component.

• Inference replicas: Minimum 2 per model, deployed across availability zones.
• Feature store replicas: Redis replication with automatic failover.
• Model registry: Store model artifacts in redundant blob storage (S3 with cross-region replication).

Retry Strategies

# Retry with exponential backoff + jitter
def predict_with_retry(request, max_retries=3):
    for attempt in range(max_retries):
        try:
            return inference_client.predict(request)
        except (NetworkError, TimeoutError) as e:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) * 0.1  # 100ms, 200ms, 400ms
            wait += random.uniform(0, wait * 0.5)  # jitter
            time.sleep(wait)

Fallback Models

A tiered fallback strategy ensures the system always returns a prediction.

# Fallback chain
def predict(request):
    for model in [primary, fallback_xgb, fallback_rules]:
        try:
            return model.predict(request)
        except:
            continue
    return DEFAULT_RESPONSE  # safe default

Circuit Breaker

Stop calling a downstream service when it is failing, allowing it time to recover.

# Circuit breaker for feature store
if circuit_breaker.state == "OPEN":
    return default_features()  # skip feature store, use cached defaults

try:
    features = feature_store.get_features(user_id)
    circuit_breaker.record_success()
    return features
except:
    circuit_breaker.record_failure()
    if circuit_breaker.failure_count > THRESHOLD:
        circuit_breaker.open()  # stop calling for 30 seconds
    return default_features()

Monitoring for Faults

• Health checks: Periodic pings to all components.
• Graceful degradation indicators: Track how often fallbacks are used.
• Automated rollback: If error rate spikes after new model deployment, automatically route traffic to previous version.