6. Scalability

Horizontal vs Vertical vs Diagonal Scaling

Approach	What It Means	Pros	Cons
Vertical (Scale Up)	Add more power to existing machines (CPU, RAM, disk)	Simple, no code changes	Hardware limits, expensive at high end, single point of failure
Horizontal (Scale Out)	Add more machines to the pool	Theoretically infinite, cost-effective, fault-tolerant	Requires load balancers, distributed coordination, code changes
Diagonal	Vertical within a tier, horizontal across tiers	Balanced approach	Complexity of both

# Horizontal scaling pattern
[Users] → [Load Balancer] → [App Server 1] → [Shared DB]
                             [App Server 2]
                             [App Server N]
# Stateless apps scale easily; stateful (DB) requires sharding/replication

Load Balancers in Depth

Beyond basic distribution, modern load balancers provide:

Health checks: Periodically ping servers; remove unhealthy ones from the pool.
SSL termination: Decrypt HTTPS traffic at the LB, forward HTTP to backends.
Rate limiting: Throttle abusive clients.
Sticky sessions: Route the same client to the same backend (via cookie or IP hash).
Circuit breaking: Stop routing to a backend when it repeatedly fails.

Auto-Scaling

Auto-scaling automatically adjusts the number of compute resources based on demand. Metrics-based (CPU > 70% → add instance) or schedule-based (scale up at 8 AM, scale down at 10 PM).

# Auto-scaling configuration (conceptual)
AutoScalingGroup:
  min: 2
  max: 20
  target_cpu: 60%
  scale_up:
    cooldown: 120s
    adjustment: +2 instances
  scale_down:
    cooldown: 300s
    adjustment: -1 instance

Sharding (Horizontal Partitioning)

Sharding splits a large database into smaller, independent databases (shards). Each shard holds a subset of the data, typically determined by a shard key.

# Shard key = user_id % 4
Shard 0: user_id % 4 == 0
Shard 1: user_id % 4 == 1
Shard 2: user_id % 4 == 2
Shard 3: user_id % 4 == 3

Challenges: Choosing a good shard key, resharding (adding more shards), cross-shard queries, distributed transactions.

CAP Theorem

A distributed system can only guarantee two of three properties simultaneously:

Consistency (C): All nodes see the same data at the same time.
Availability (A): Every request receives a response (success or failure).
Partition Tolerance (P): The system continues to operate despite network failures.

System design pillars showing trade-offs between consistency, availability, and partition tolerance

In practice, network partitions will happen, so you choose between CP and AP. CA is not possible in distributed systems.

Consistency Models

Strong consistency: All reads see the latest write. Hard to achieve at scale.
Eventual consistency: Given enough time, all replicas converge. Common in DNS, CDNs, NoSQL.
Read-your-writes: A user sees their own writes immediately, but others may not.
Monotonic reads: A user never sees an older version after seeing a newer one.

Exercise: Design a sharding strategy for a social media platform where users can follow others and see a timeline. Describe your shard key choice, how you handle cross-shard queries (e.g., fetching a timeline from followed users), and how you would reshard from 4 to 8 shards with zero downtime.