LLM benchmark
A public, standardized test set used to compare models with each other, as opposed to evals built for your own task.
Benchmarks such as MMLU or SWE-bench give every model the same tasks and publish a leaderboard. They are useful for a shortlist, but they measure general ability, not your documents, language or edge cases. Scores also inflate over time, as test questions leak into training data and vendors tune for the leaderboard. Your own evals answer the question a benchmark cannot: which model works for this task at this cost.
Related terms
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- Golden dataset: A fixed set of real inputs with expected answers or checks, used to test an AI feature after every prompt or model change.
- Model routing and fallback: Sending each request to the model that fits it, and to another one when the first fails.
- Cost per task (agent unit economics): What one finished agent task really costs: tokens, retries, tool calls and the human time to review it, compared with doing it another way.
