20. Trade-Offs

Every architectural decision in AI system design involves a trade-off. Recognizing and articulating these trade-offs is the mark of a senior engineer.

Accuracy vs Latency

The most fundamental trade-off. Larger, more accurate models are slower and more expensive.

Decision	Accuracy	Latency	Cost
Large model (e.g., GPT-4)	Best	~2s per response	High
Distilled model (e.g., GPT-4 → Llama-7B)	Good (90% of large)	~100ms	Low
Quantized model (INT8)	98% of FP16	4x faster	Lower
Rule-based fallback	Moderate	~5ms	Minimal

How to decide: If your use case tolerates occasional lower quality (e.g., product recommendations), optimize for latency. If quality is critical (e.g., medical diagnosis), use a larger model with a longer timeout.

Batch vs Real-Time

Factor	Batch	Real-Time
Freshness	Hours—days stale	Sub-second
Infrastructure complexity	Lower	Higher (stream processing, low-latency DB)
Cost	Lower (use spot instances, batch optimization)	Higher (always-on, low-latency infra)
Error handling	Simple (retry entire batch)	Complex (partial failures, exactly-once processing)

Hybrid pattern: Train models in batch (offline, comprehensive). Serve predictions in real-time (online, fast). Update features via micro-batch for near-real-time freshness.

Autonomy vs Control

In agentic AI systems, how much autonomy do agents have?

High autonomy: Agents decide tools, plans, and actions independently. Faster but riskier (hallucinations, incorrect actions).
High control: Human approval required for critical actions. Safer but slower. Suitable for high-stakes domains (finance, healthcare).
Graduated autonomy: Low-risk actions are automatic. High-risk actions require approval. The best balance for most systems.

Cost vs Quality

Every optimization has a price. A 1% accuracy improvement might cost 10x more in GPU training time. A 10ms latency reduction might require expensive hardware.

# Cost-quality decision framework
For each optimization:
  1. What is the cost? (infra, engineering time, complexity)
  2. What is the benefit? (business metric: CTR, conversion, revenue)
  3. What is the ROI? (benefit / cost)
  4. Is there a simpler alternative?

Example:
  Optimization: Add 2 more models to ensemble
  Cost: +$500/month GPU, +20ms latency
  Benefit: +0.5% accuracy
  ROI: $500/month for 0.5% — only worth it if 0.5% = $5000+ in revenue

Consistency vs Availability (CAP for AI)

Strong consistency: All feature updates visible immediately. Harder to scale. Use for payment ML models.
Eventual consistency: Features may be stale. Easier to scale. Use for recommendations, search ranking.

Model Complexity vs Interpretability

Model Type	Accuracy	Interpretability	Use Case
Linear regression / Logistic	Low—Moderate	High	Credit scoring (regulated)
Decision Trees / Random Forest	Moderate—High	Moderate	Fraud detection
XGBoost / LightGBM	High	Low (SHAP needed)	Ranking, CTR prediction
Deep Neural Networks	Highest	Very Low	Image, text, complex patterns
LLMs	Highest	Low (emergent)	Chat, generation, reasoning

Exercise: Your team is building an AI system that automatically approves or rejects small business loan applications. List 5 significant trade-offs you must navigate. For each: describe the two options, the pros/cons of each, and your recommended choice with justification. Consider: accuracy vs speed, batch vs real-time, model complexity vs regulatory requirements, autonomy vs human review, cost vs quality.