Part 9
As evaluation grows from a script into a discipline, it needs infrastructure: pipelines that run automatically, versioned data, tracked experiments, and views that turn results into decisions. This part covers the pieces that make evaluation repeatable, auditable, and scalable.
An evaluation pipeline automates the whole flow: load the dataset, run the model, compute metrics, store results, and publish a report. Build it so a single command (or a CI trigger) reproduces any past evaluation. Treat the pipeline as code — reviewed, versioned, and tested — so a "better metric" one month does not silently change how old results were computed.
Benchmark datasets change: bugs get fixed, examples get added, duplicates get removed. Without versioning you cannot tell whether a score change came from the model or from the data. Store datasets in a versioned store (like DVC, Git-LFS, or a database with versioning), record the dataset version alongside every evaluation result, and never overwrite a released dataset in place.
Experiment tracking records every evaluation run: the code version, the dataset version, the model and parameters, the metrics, and the artifacts. Tools like MLflow or Weights & Biases provide this. The payoff is a searchable history — "what was the best answer-relevance we ever got, and with which prompt?" — instead of a pile of unnamed CSV files.
import mlflow
with mlflow.start_run(run_name="rag-v2"):
mlflow.log_params({"embedding": "bge-small", "top_k": 5})
mlflow.log_metrics({"context_precision": 0.83, "faithfulness": 0.91})
mlflow.log_artifact("results.json")Batch evaluation processes large datasets efficiently by batching calls and parallelizing. LLM evaluation is slow and rate-limited, so batching thousands of examples matters. Store results incrementally and make runs resumable — if a batch fails midway, restart from the checkpoint instead of the beginning.
When one machine is not enough, split the evaluation across many workers. Design for statelessness: each worker pulls a shard of examples, runs the model, and writes results back, with retries on transient failures. For LLM calls, a shared result store plus idempotent writes lets you scale horizontally without double-counting work.
Dashboards visualize metrics and trends so the team can see regressions before users do. Design them around decisions: a main view with the metrics that gate releases, a drill-down for debugging a specific change, and time-series views that reveal slow drift. Keep dashboards honest — show error bars on noisy estimates instead of a single over-precise number.
Leaderboards rank models or configurations against a fixed set of benchmarks. They are powerful for choosing models and tracking progress, and dangerous when over-trusted: models overfit to public leaderboards, and one aggregate number hides per-slice failures. Publish per-metric breakdowns and require that entries report the evaluation conditions, not just the headline score.