1. Evaluation Fundamentals

Part 1

Before you measure anything, you need to know what "good" means for your system. This part lays the groundwork: what evaluation is, how business goals differ from metrics, how to make results trustworthy, and how to design experiments and benchmark datasets you can rely on.

What is AI Evaluation

AI evaluation is the disciplined practice of measuring how well an AI system performs. You define what you care about, collect evidence (model outputs, user feedback, benchmark results), and compare that evidence against a target. The output of evaluation is information you can act on: which changes improve the system, which regressions you introduced, and whether the system is ready to ship.

Evaluation is not a one-time step. It happens during development, before release, and continuously after deployment. A good evaluation answers a question like "Did the new prompt make answers more accurate?" rather than a vague "Is the model good?"

Key idea: Evaluation converts subjective impressions into measurable, comparable evidence. Without it, you are guessing.

Metrics vs Objectives

A common mistake is confusing a metric with an objective. An objective is a business or product goal — "customers trust our support answers" or "users complete sign-up without frustration." A metric is a measurable number you track as a proxy for that goal — "the percentage of answers a human expert rated as correct."

Objective	Example Metric
Customers trust support answers	Answer accuracy, citation accuracy
Users find answers quickly	Time to first useful answer, search top-3 hit rate
Fewer support tickets	Ticket deflection rate
Safe experience for everyone	Toxicity rate, refusal-to-unsafe-request rate

The relationship is rarely perfect. Metrics can be gamed, and an objective can improve while a proxy metric stays flat. Whenever you pick a metric, ask: "If this number goes up, does the user experience actually get better?" If not, the metric is the wrong proxy.

Reliability & Validity

Two qualities separate trustworthy evaluation from noise:

Reliability — the measurement is consistent. Run the same evaluation twice and you get roughly the same result. Unreliable causes include small samples, random seeds, non-deterministic model output, and human raters who disagree with each other.
Validity — the measurement actually captures what you intend. A quiz score that only measures memorization is not a valid measure of understanding. A metric can be reliable yet invalid, so check both.

For human ratings, reliability is often quantified with inter-rater agreement measures such as Cohen's kappa or Krippendorff's alpha. Low agreement tells you your rating instructions are ambiguous — fix the rubric before trusting the scores.

Statistical Testing

When you compare two model versions, the difference you see might be chance. Statistical testing tells you how confident you can be that a difference is real.

Null hypothesis: both versions perform equally; any observed gap is random noise.
p-value: the probability of seeing a difference at least this large if the null hypothesis were true. A common threshold is p < 0.05.
Confidence interval: the range where the true difference probably lies. If the interval excludes zero, the difference is statistically significant.

Common tools: paired tests (like the paired t-test or Wilcoxon signed-rank test) work well when the same examples are scored under both versions. For LLM outputs, bootstrap resampling is popular because it needs few assumptions.

import numpy as np
scores_a = [0.92, 0.88, 0.95, 0.81, 0.90]
scores_b = [0.94, 0.91, 0.93, 0.85, 0.96]

# Paired differences
diffs = np.array(scores_b) - np.array(scores_a)
mean, sd = diffs.mean(), diffs.std(ddof=1)
n = len(diffs)
se = sd / np.sqrt(n)
print(f"mean diff = {mean:.3f} +/- {1.96*se:.3f} (95% CI)")

Beware: with thousands of examples, tiny differences become "significant" even if they are useless in practice. Always ask whether the effect is large enough to matter, not just statistically detectable.

Experimental Design

A reliable experiment controls for everything except the change you are testing. Design principles to follow:

Control group: always run a baseline (the current version) alongside your candidate.
Fixed evaluation set: use the same examples for both versions so differences come from the models, not the data.
Blind scoring: when humans grade outputs, hide which version produced each answer to avoid bias.
Randomization: assign examples to raters or treatment groups randomly to spread out confounders.
Replication: run multiple seeds or repeated trials to estimate variance.

For online experiments, split traffic so that each user sees one consistent version — switching versions mid-session contaminates the comparison. Plan the minimum sample size before you start so the experiment can actually detect the effect size you care about.

Golden Datasets

A golden dataset (also called a golden test set or benchmark) is a curated collection of examples with known, trusted answers. It is the fixed yardstick you measure against across versions and over time.

Build one well:

Cover the real workload: include the question types and difficulty levels your users actually produce.
Enough volume: a few dozen examples give noisy estimates; aim for hundreds or thousands where feasible.
Expert-verified answers: answers should be reviewed by people who know the domain, not just one author.
Version control: treat the dataset like code — changes go through review, and you can compare old vs new scores.
Guard against leakage: never let training or few-shot examples appear in the golden set.

Practice Task: Pick a task you work on (chat answers, search, classification). Write ten representative examples with gold answers, then define two metrics: one objective proxy and one directly tied to user experience. Explain in one sentence how the metric could mislead you.