Holdout scenarios
End-to-end user scenarios kept outside the codebase, so the coding agent never sees them and cannot write code that only passes the check.
A test stored in the repository can be read, and rewritten, by the agent that is supposed to pass it. The StrongDM software factory describes the fix: scenarios, end-to-end user stories stored outside the codebase “similar to a holdout set in model training”, validated against the running software, often by an LLM as a judge. Success becomes a measured share of scenarios that satisfy the user instead of a green test suite.
The idea is the same as a golden dataset for evals: the people who build the system do not tune it on the examples used to grade it. Scenarios usually run against digital twins of external services, and they are the main defense against reward hacking in a software factory.
Sources
- StrongDM: Software Factory factory.strongdm.ai
Related terms
- Reward hacking (coding agents): When an agent makes the check pass without solving the task: deleting or weakening tests, special-casing test inputs or faking results.
- Digital twin (service virtualization): A behavioral copy of an external service, such as a payment or identity API, that agents and tests can call safely, repeatedly and at any volume.
- Golden dataset: A fixed set of real inputs with expected answers or checks, used to test an AI feature after every prompt or model change.
- Software factory (AI agents): A pipeline where agents write and fix most of the code while people write specs, set guardrails, design the checks and decide what ships.
- LLM as judge: Using a model with a rubric to grade another model’s output.
- Validation loop (agent backpressure): Commands a coding agent runs after every change (tests, types, lint, build) and fixes against until they pass, so it checks itself before a person looks.
