Ragas
An open-source Python library for evaluating RAG systems with metrics such as faithfulness, answer relevancy, context precision and context recall.
Ragas came out of a 2023 research paper and became a common reference point for RAG metrics; other tools implement the same ones. It scores retrieval (context precision, context recall) separately from generation (faithfulness, answer relevancy), mostly using a judge model, and can generate test questions from your documents. Scores depend on the judge model and its prompts, so they are comparable between your own runs, not between vendors.
Related terms
- Faithfulness (groundedness): A RAG metric: the share of claims in an answer that are supported by the retrieved sources, not by the model’s memory or invention.
- Answer relevancy: A RAG metric that scores how directly an answer addresses the question asked, regardless of whether it is factually correct.
- Context precision: A RAG retrieval metric: whether the passages relevant to the question are ranked at the top of what was retrieved.
- Context recall: A RAG retrieval metric: how much of the information needed for the correct answer was actually retrieved.
- DeepEval: An open-source Python framework that tests LLM applications the way pytest tests code, with ready-made metrics for RAG and agents.
- Promptfoo: An open-source command-line tool for testing prompts and models against test cases defined in a config file, including security red teaming.
