AI red teaming
Deliberately attacking your own AI feature with harmful, tricky and injected inputs to find failures before users or attackers do.
A red team writes prompts that try jailbreaks, injections hidden in documents, requests for other users’ data, and tool calls with dangerous arguments, then records what got through. Unlike a classic pentest, results vary between runs, so the useful output is a set of attack cases that become permanent evals run after every model or prompt change.
Related terms
- Jailbreak (LLM): A prompt written to make a model ignore its safety rules or its system prompt, sent by the user themselves.
- Prompt injection: Instructions hidden in data (a web page, an email, a file) that the model follows as if they came from you.
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- Penetration test (pentest): An authorized, time-boxed attack on your app by security testers, ending in a report of what they could break and how badly.
