Data and compute sharding are essential when datasets or model parameters exceed single-machine capacity.
Data sharding: Partition training data across workers. Each worker trains on a subset; gradients are averaged (all-reduce).
Model sharding: Split model layers across GPUs (pipeline parallelism).
Tensor sharding: Split individual tensors across devices (tensor parallelism). Used in training large language models.
# Data Parallelism (simplified)
Worker 1: sees data[0:10K], computes gradients G1
Worker 2: sees data[10K:20K], computes gradients G2
Worker N: sees data[...], computes gradients GN
All-reduce: G = (G1 + G2 + ... + GN) / N
All workers apply G to their model copy
Distributed Training
Strategy
Model Size
Communication
Frameworks
Data Parallelism
Fits on one GPU
Gradients (frequent)
PyTorch DDP, Horovod
Pipeline Parallelism
Fits on a few GPUs
Activations (moderate)
DeepSpeed, Megatron-LM
Tensor Parallelism
Very large (100B+ params)
Intermediate tensors (intra-node)
Megatron-LM, FSDP
FSDP (Fully Sharded)
Large (1B—100B params)
All-gather (efficient)
PyTorch FSDP
Model Compression
Smaller models mean lower latency and lower cost:
Quantization: FP32 → FP16 (2x speed), INT8 (4x speed). Lossy but often negligible quality drop.
Pruning: Remove low-magnitude weights. Can reduce model size by 50—90% with fine-tuning.
Distillation: Train a small "student" model to mimic a large "teacher" model.
LoRA adapters: Fine-tune small adapter matrices instead of the full model. Multiple adapters can be served from one base model.
# Quantization impact
Model: Llama-7B
FP32: 28 GB GPU memory, 50ms per token
FP16: 14 GB GPU memory, 28ms per token
INT8: 7 GB GPU memory, 15ms per token
Inference Load Balancing
Request-level: Round-robin across inference replicas.
Model-level: Route requests to different model versions (canary, shadow).
GPU-level: Dynamic batching groups multiple requests into one inference call for GPU efficiency.
Auto-scaling: Scale number of inference pods based on queue depth or CPU/GPU utilization.
Cost-Performance Trade-offs
Every optimization decision involves a cost-benefit analysis:
Technique
Latency Improvement
Cost Impact
Quality Impact
Quantization (FP16)
~2x
Lower (less GPU time)
Minimal
Quantization (INT8)
~4x
Lower
Small (0.5—2%)
Distillation (10% size)
~10x
Much lower
Moderate (3—5%)
Dynamic batching
Higher throughput
Better GPU utilization
No impact
Exercise: You need to serve a 70B parameter LLM at 100 requests/second with p99 latency under 1 second. You have 8 A100 GPUs (80GB each). Design the serving architecture: model parallelism strategy, quantization level, batch size, and number of replicas. Show your calculations.