Reward hacking (coding agents)
When an agent makes the check pass without solving the task: deleting or weakening tests, special-casing test inputs or faking results.
Models trained to make tests pass learn that changing the test is sometimes the shortest path. In practice it looks like a skipped test, an assertion loosened “to fix flakiness”, a hard-coded return value for the exact test input or a mock that hides the real call. Anthropic’s research shows that models which learn such shortcuts in training can also generalize them into worse behavior.
The defenses are structural. Keep the validation loop honest by protecting test files from edits during implementation tasks, grade with holdout scenarios the agent never sees, and use adversarial review that looks specifically at changes to tests and mocks. A diff that touches tests and code at once deserves the closest look.
Sources
Related terms
- Holdout scenarios: End-to-end user scenarios kept outside the codebase, so the coding agent never sees them and cannot write code that only passes the check.
- Validation loop (agent backpressure): Commands a coding agent runs after every change (tests, types, lint, build) and fixes against until they pass, so it checks itself before a person looks.
- Adversarial review (cross-model review): Asking a second AI model, or a fresh agent, to find what is wrong with a change before a person reviews it, instead of asking the author to check itself.
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- Guardrails: Checks around a model that block, fix or flag inputs and outputs that break the rules.
