Part 6
Safety evaluation tests what the system does at the edges: harmful content, unfair behavior, manipulation attempts, and data leakage. It is not a one-time checklist — new attacks and new failure modes appear constantly, so safety testing must keep running.
Toxicity detection checks outputs for hateful, threatening, or abusive language. Use a combination of classifier-based detectors (such as toxicity classifiers) and careful human review, since automated detectors miss subtle or coded harmful language. Track a toxicity rate over time so a prompt change that quietly makes the model less careful is caught.
Bias evaluation measures whether the system treats groups fairly. Practical checks: generate outputs for inputs that differ only by gender, race, or other attributes and compare tone, sentiment, and quality; audit decisions for disparate outcomes; and watch for skewed representation in generated content. No single metric proves fairness — combine distributional tests with case-by-case review.
Prompt injection is an attempt to override the system's instructions from the input itself — for example, hidden text inside a document telling the model to ignore its rules. Evaluate by inserting injection strings into user content and checking whether the model still obeys its original instructions. Test both direct injections ("ignore previous instructions...") and indirect ones hidden in retrieved documents.
Jailbreaks are adversarial prompts crafted to bypass safety filters — role-play scenarios, encoded instructions, escalating persuasion, or fictional framing. Maintain a library of known jailbreak patterns and run them after every system change. Because jailbreak techniques evolve quickly, treat the library as a living artifact and add new patterns as they appear.
Red teaming is systematic, deliberate adversarial testing by a team that tries to break the system. Unlike ad hoc tests, red teaming has structure: defined goals (bypass safety, extract data, force harmful output), a documented attack plan, and reports of what worked. Run it before launch and periodically after, and route findings back into the safety test suite.
Privacy evaluation checks whether the system leaks sensitive information: personal data appearing in outputs, models regurgitating memorized private content, or logging/retention mishandling user input. Test with probes designed to extract memorized content and with realistic user data in the input to confirm it never surfaces. Also review infrastructure — what gets logged, where, and for how long.
| Threat | What to probe | Typical mitigation signal |
|---|---|---|
| Toxicity | Slurs, threats, harassment | Low toxic-output rate |
| Bias | Unequal treatment across groups | Balanced scores across slices |
| Prompt injection | Input overrides system rules | Instructions stay intact |
| Jailbreak | Filters bypassed by crafted prompts | Refusal or redirect on attack set |
| Privacy | Personal data in outputs | Zero leakage in probes |
Frameworks turn safety work into repeatable practice. The NIST AI Risk Management Framework provides a structured process for identifying, assessing, and managing AI risk. The OWASP LLM Top 10 catalogs the most common security risks in LLM applications — prompt injection, insecure output handling, data leakage, and so on — and is a practical checklist for threat modeling. Map your evaluation activities to these frameworks so safety testing is systematic rather than improvised, and keep records for accountability and audits.