9. Reliability

High Availability (HA)

Availability measures the percentage of time a system is operational. Standard targets:

• 99.9% ("three nines") → ~8.7 hours downtime/year
• 99.99% ("four nines") → ~52 minutes downtime/year
• 99.999% ("five nines") → ~5 minutes downtime/year

HA is achieved through redundancy — eliminating single points of failure by having backup components ready to take over.

# Active-Passive HA
Primary server handles traffic
Standby server mirrors primary (data replication)
If primary fails → health check timeout → failover to standby
Users experience brief interruption (seconds to minutes)

# Active-Active HA
Both servers handle traffic simultaneously
If one fails, the other continues at reduced capacity
Zero failover time, but requires careful conflict resolution

Fault Tolerance

A fault-tolerant system continues operating, possibly at reduced capacity, when components fail.

Techniques

• Redundancy: Multiple instances of critical components (servers, disks, network paths).
• Graceful degradation: Disable non-critical features during overload (e.g., Netflix's "recommendations degraded" banner).
• Circuit breaker: When a downstream service fails repeatedly, stop calling it and return a fallback response.
• Bulkhead: Isolate components into separate pools so failure in one does not exhaust shared resources (like ship compartments).
• Retry with backoff: Automatically retry failed operations with exponential delay.
• Dead letter queue: Store messages that cannot be processed for manual inspection.

# Circuit Breaker Pattern (Pseudocode)
if circuit.state == CLOSED:
    try: result = call_service(); circuit.record_success()
    except: circuit.record_failure()
elif circuit.state == OPEN:
    return fallback_response() # Don't even try
elif circuit.state == HALF_OPEN:
    result = call_service() # Probing request
    if success: circuit.reset() else: circuit.open_again()

Backup Strategies

RPO: Maximum acceptable data loss (in time). RTO: Maximum acceptable downtime to restore.

Disaster Recovery (DR)

• Backup & Restore: Back up to secondary region; restore if primary fails. Cheapest, highest RTO.
• Pilot Light: Core services (DB, critical config) are always on in secondary region; scale up when DR triggers.
• Warm Standby: A scaled-down version of production runs in secondary region; scale up on failover.
• Multi-Site Active-Active: Fully operational in both regions; traffic load-balanced. Most expensive, lowest RTO.

SLA, SLO, SLI

• SLI (Service Level Indicator): A quantifiable metric (e.g., request latency p99 < 200ms).
• SLO (Service Level Objective): Target value for SLI (e.g., 99.9% of requests under 200ms).
• SLA (Service Level Agreement): Contract with customer backed by penalties (e.g., 99.99% uptime guarantee).