10. Practical Projects

Part 10

The fastest way to learn evaluation is to build it. These seven projects move from a simple metric library up to a production monitoring dashboard. Each builds on the parts before it, and each is small enough to finish in a focused weekend.

Project 1 · ML Evaluation Library

Implement the classic metrics from Part 2 in pure Python — accuracy, precision, recall, F1, MAE, RMSE, R², and the ranking metrics MRR and NDCG — without any ML library. Write unit tests with tiny hand-computable examples. Doing this by hand makes every formula concrete and gives you a test double for the bigger projects.

Project 2 · LLM Judge System

Build an automated judge: a prompt template that scores answers against a rubric, a loop that runs it over a golden dataset, and an aggregation step that reports average scores plus agreement with a small set of human labels. Include the pairwise-comparison mode from Part 3. This is the core engine you will reuse in every later project.

Project 3 · RAG Evaluation Pipeline

Wire together a small RAG app — an index, a retriever, and a generator — with evaluation on both halves. Compute retrieval metrics on your labeled query set and faithfulness/groundedness on the generated answers. Add a reranker and measure whether NDCG improves enough to justify the latency cost.

Project 4 · AI Agent Evaluator

Run an agent on a set of scripted tasks and score the trajectory: task success, tool-call validity, and wasted steps. Store each run's full log (thoughts, tool calls, observations) so a failing score is always diagnosable. Start with a sandboxed task like file manipulation or API calls so failures are harmless.

Project 5 · Safety Benchmark Suite

Assemble an attack library from Part 6 — toxicity probes, bias probes, injections, jailbreaks, privacy extractions — and run it as a repeatable suite. Score each case blocked/partial/failed, keep a severity rating, and publish a simple pass rate per category. Version the suite and add one new pattern each week to keep it alive.

Project 6 · Continuous Assessment CI/CD

Hook evaluation into CI so every PR to a prompt, model config, or RAG pipeline runs the golden-set suite automatically. Define a quality gate: the build fails if any core metric drops by more than a tolerance. This is where evaluation becomes a team practice instead of a personal habit. Start with one repository and the metrics you trust most.

# .github/workflows/eval.yml (sketch)
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python run_eval.py --dataset golden-v3
      - run: python check_gate.py --drop-tolerance 0.02

Project 7 · Production Evaluation Dashboard

Build a dashboard that merges offline results, live feedback, and system health — the metric hierarchy from Part 8. Track the three families (outcome, quality, health) over time, show drift alerts, and link each metric to the log query that explains it. Even a simple static dashboard that regenerates daily beats no dashboard at all.