Prompt regression
An answer that used to be right goes wrong after a change to the prompt or the model.
Prompts are edited to fix one answer and quietly change others: a rule gets lost in a rewrite, a "be shorter" line drops a required detail, a cheaper model skips a step. Because model output varies between runs, one failed run is not proof: run each case several times and treat a case that passes sometimes as unstable, not broken.
Related terms
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- LLM as judge: Using a model with a rubric to grade another model’s output.
- Model routing and fallback: Sending each request to the model that fits it, and to another one when the first fails.
