15. Fault Tolerance

AI systems must handle failures gracefully. Unlike traditional services where a 500 error is acceptable, AI systems often serve critical decisions (fraud detection, medical triage, autonomous driving) where failing open or returning a wrong answer has real consequences.

Redundancy

Deploy multiple replicas of every component across availability zones or regions.

• Inference replicas: Minimum 2+ behind a load balancer. If one fails, traffic shifts to others.
• Feature store replicas: Redis cluster with replicas. Read replicas serve if primary is slow.
• Cross-region: Active/passive or active/active across regions for disaster recovery.

Retry Strategies

Transient failures (network blips, DB timeouts, GPU OOM) should be retried. But naive retries can overload the system.

# Exponential backoff with jitter
def retry_with_backoff(fn, max_retries=3):
    for attempt in range(max_retries):
        try:
            return fn()
        except TransientError as e:
            if attempt == max_retries - 1:
                raise  # final attempt failed
            wait = (2 ** attempt) * 0.1  # 100ms, 200ms, 400ms
            wait += random.uniform(0, 0.05)  # jitter
            time.sleep(wait)

Circuit Breakers

When a downstream service (e.g., feature store, embedding API) is consistently failing, stop calling it to prevent cascading failures.

# Circuit breaker states for AI inference
CLOSED: Normal operation. Requests flow to feature store.
  If 5 consecutive failures → transition to OPEN

OPEN: Requests bypass feature store. Use default features.
  After 30 seconds → transition to HALF_OPEN

HALF_OPEN: Allow 1 test request to feature store.
  If success → transition to CLOSED
  If failure → transition to OPEN, wait longer (exponential backoff)

Fallback Models

When the primary model is unavailable (GPU failure, model corruption, timeout), serve a fallback.

# Fallback chain logic
def predict(request):
    try:
        return primary_model.predict(request)  # 100ms
    except ModelError:
        try:
            return fallback_model.predict(request)  # 25ms
        except ModelError:
            try:
                return rule_based_predict(request)  # 5ms
            except:
                return DEFAULT_RESPONSE  # safe default

Graceful Degradation

When parts of the system fail, degrade features rather than failing entirely:

• No personalization: Use generic recommendations instead of personalized ones.
• No real-time features: Fall back to cached batch features.
• No embedding model: Fall back to keyword search.
• No GPU: Fall back to CPU (slower but functional).