The first step in any AI system design is to clearly define the problem. Without this, you risk building a system that solves the wrong thing or fails to meet real-world constraints.
Ask these questions before designing:
What is the business problem? "Fraud detection" vs "reduce chargeback rate by 30%" — the second is measurable.
Who are the users? Internal analysts? External customers? Automated systems?
What is the input and output? Raw transaction data → fraud probability score. Free-text query → answer with citations.
What is the decision boundary? At what confidence threshold do we act?
What are the success criteria? Accuracy, latency, throughput, uptime?
Clarifying Constraints
Every system operates within constraints. Identify them early:
Constraint Type
Examples
Latency
API must respond within 200ms p99
Throughput
Handle 10,000 requests/second
Cost
Inference budget <$0.001 per request
Data Governance
Must comply with GDPR — data cannot leave EU region
Hardware
Must run on CPU (no GPU), 4GB RAM
Model Size
Model must fit in 500MB for edge deployment
Functional vs Non-Functional Requirements
# Functional Requirements (what the system does)
- Accept transaction data via REST API
- Return fraud probability score (0.0—1.0)
- Log all predictions for audit trail
- Allow admin to view prediction history
# Non-Functional Requirements (how the system performs)
- p99 latency < 200ms
- 99.95% uptime
- Handle 5,000 QPS peak
- Encrypt all data at rest and in transit
- Support canary deployments for model rollouts
Key insight: Non-functional requirements drive most architectural decisions. Latency determines caching and model size. Uptime determines redundancy and failover. Cost determines whether you use GPU or CPU, on-prem or cloud.
Avoiding Common Pitfalls
Scope creep: "Let's also add sentiment analysis" — stick to the primary problem.
Over-engineering: Don't design for 100M users if you have 10K.
Ignoring data pipelines: Most AI failures are data failures, not model failures.
Starting with the model: Start with the problem, not the solution.
Exercise: You are asked to design an AI customer support chatbot for an e-commerce platform. Write down: a) 3 functional requirements, b) 3 non-functional requirements, c) 3 constraints, d) 1 measurable success criterion. Be specific with numbers where possible.