Trajectory evals
Tests that grade an agent’s whole path to an answer: which tools it called, in what order, how many steps and at what cost, not only the final output.
Two agents can reach the same correct answer while one reads a customer record it had no reason to touch and takes five times the steps. Final-answer evals miss that. A trajectory eval checks the recorded run against expectations: required and forbidden tool calls, step and cost limits, and sometimes an LLM judge for the reasoning. The traces come from the same observability you use in production.
Related terms
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- LLM as judge: Using a model with a rubric to grade another model’s output.
- Golden dataset: A fixed set of real inputs with expected answers or checks, used to test an AI feature after every prompt or model change.
- LLM observability: Tracing every model call with inputs, outputs, latency, tokens, cost and the user outcome.
- Cost per task (agent unit economics): What one finished agent task really costs: tokens, retries, tool calls and the human time to review it, compared with doing it another way.
