11. Scalability

Horizontal Scaling

Add more machines to distribute the load. The standard approach for stateless inference servers.

# Horizontal scaling for inference
Load balancer distributes requests to N replicas
All replicas identical, stateless (model loaded from shared storage)
Auto-scaling based on:
  - CPU/GPU utilization (>70% → add replica)
  - Queue depth (>100 pending requests → add replica)
  - Request latency (p99 > 200ms → add replica)

Partitioning (Sharding)

For stateful components (feature stores, model registry), partition data across nodes.

• Hash-based sharding: shard = hash(entity_id) % N. Even distribution, but resharding is expensive.
• Range-based sharding: user_id 1—1M on shard A, 1M—2M on shard B. Good for range queries, risk of hot spots.
• Directory-based sharding: Lookup table maps keys to shards. Flexible but adds latency.

Queues for Load Smoothing

Use message queues to decouple request arrival from processing. Queues absorb traffic spikes and provide back-pressure.

# Queue-based inference
Request → API Gateway → Request Queue (Kafka/SQS)
Inference workers consume from queue:
  - Pull batch of requests (up to 32)
  - Run batch inference (GPU)
  - Write results to response topic
  - Consumer reads response and returns to client

Benefits:
  - Burst absorption (10K requests arrive, queue buffers)
  - Worker auto-scaling (more workers = faster drain)
  - Retry on failure (messages return to queue)

Auto-Scaling Strategies

Load Balancing Algorithms for ML

• Round-robin: Simple, works for equal-capacity servers.
• Least connections: Prefer server with fewest active requests. Good for variable inference times.
• Weighted: Route more traffic to GPU-enabled servers, less to CPU-only.
• Consistent hashing: Route same user to same server. Useful for model-level caching.

Scaling the Data Pipeline

• Partition Kafka topics by entity key for ordered processing.
• Scale Spark/Dask workers based on data volume.
• Feature store read replicas: Multiple Redis replicas for read-heavy serving.
• Model registry: CDN for model binary downloads (avoid overloading registry).