Human evaluation
People, ideally domain experts, grading AI outputs against a written rubric; the reference that automated metrics are checked against.
Automated scores are only as good as their agreement with human judgment, so a sample graded by people is how you calibrate an LLM as judge. Two things make it reliable: a rubric specific enough that different graders reach the same verdict, and measuring that agreement (inter-annotator agreement). When graders disagree often, the rubric is unclear, and no metric built on it will be stable.
Related terms
- LLM as judge: Using a model with a rubric to grade another model’s output.
- Ground truth: The known correct answers or labels that a model’s output is compared against in an evaluation.
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- Human in the loop: A person approves, edits or rejects the agent’s action before it takes effect.
