DeepEval
An open-source Python framework that tests LLM applications the way pytest tests code, with ready-made metrics for RAG and agents.
A test case holds the input, the actual output and, where needed, the expected output and the retrieved context; metrics score it and the test passes above a threshold. It ships RAG metrics similar to those in Ragas and G-Eval, a metric where a judge model scores against criteria you write in plain language. It is developed by Confident AI, which sells a hosted platform on top of it.
Related terms
- Ragas: An open-source Python library for evaluating RAG systems with metrics such as faithfulness, answer relevancy, context precision and context recall.
- Promptfoo: An open-source command-line tool for testing prompts and models against test cases defined in a config file, including security red teaming.
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- LLM as judge: Using a model with a rubric to grade another model’s output.
