Retrieval-Augmented Generation (RAG) Evaluation
Lab Playbook 3
Test context relevance, faithfulness, recall metrics, and answers grounding using DeepEval.
On Day 2, we evaluated unstructured text using LLM-as-a-Judge. On Day 3, we specifically focus on RAG (Retrieval-Augmented Generation) applications. Evaluating RAG systems requires testing three critical aspects: whether the retrieved context is relevant, whether the model's answer is faithful to the context (no hallucination), and whether the final answer is relevant to the original user query. We will learn how to configure DeepEval to measure context recall and faithfulness metrics.
The official reference codebase for Day 3 is hosted in the repository. Clone the repository and navigate to day-3-rag-evaluation-deepeval to run the working solution.
git clone https://github.com/letstrnsfrm-ai/data-quality-llm-testing-roadmap.git
cd data-quality-llm-testing-roadmap/day-3-rag-evaluation-deepevalDeepEval Setup & RAG Parameters (QE Setup)
Quality Engineers begin by installing the necessary evaluation frameworks and setting up the evaluation parameters.
pip install deepeval
deepeval loginYou have configured the Python testing environment and installed DeepEval for RAG auditing.
RAG Evaluation Prompting (Orchestrator Role)
Direct the agent to write a template prompting the RAG evaluation framework using custom metrics.
Please construct a Python snippet using DeepEval's FaithfulnessMetric and AnswerRelevanceMetric.
1. The script should define a LLMTestCase representing a typical user interaction.
2. The user query is: "How do I reset my password?"
3. The retrieved context is: "To reset your password, visit the security settings page and click 'Forgot Password'. You must verify your email within 10 minutes."
4. The candidate response is: "You can reset your password by going to the security settings page, but make sure to verify your email within 10 minutes."
5. Configure the faithfulness metric to evaluate alignment with retrieved context.Here is the Python setup to evaluate the RAG metrics using DeepEval:
```python
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevanceMetric
# Define the test case
test_case = LLMTestCase(
input="How do I reset my password?",
actual_output="You can reset your password by going to the security settings page, but make sure to verify your email within 10 minutes.",
retrieved_contexts=["To reset your password, visit the security settings page and click 'Forgot Password'. You must verify your email within 10 minutes."]
)
# Instantiate the metrics
faithfulness_metric = FaithfulnessMetric(threshold=0.7)
relevance_metric = AnswerRelevanceMetric(threshold=0.7)
```
Retrieval & Grounding Validation (Orchestrator Role)
Quality Engineers validate the retrieval accuracy and grounding scores using mock evaluation pipelines.
Write a script 'test_rag_evaluation.py' that imports DeepEval's metrics, runs the evaluation on our test case, and prints the resulting faithfulness and relevance scores.Here is the evaluation script ('test_rag_evaluation.py'):
```python
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
def test_rag_faithfulness():
test_case = LLMTestCase(
input="How do I reset my password?",
actual_output="You can reset your password by going to the security settings page, but make sure to verify your email within 10 minutes.",
retrieved_contexts=["To reset your password, visit the security settings page and click 'Forgot Password'. You must verify your email within 10 minutes."]
)
metric = FaithfulnessMetric(threshold=0.7)
metric.measure(test_case)
print(f"Faithfulness Score: {metric.score}")
print(f"Reasoning: {metric.reason}")
assert metric.is_successful(), "Faithfulness score is below the threshold!"
```
Execution Loop & Faithfulness Gates (Orchestration Loop)
Embed the evaluation within the orchestration loop to reject hallucinatory outputs dynamically.
Modify our python runner to execute the DeepEval test runner commands, and verify that the build fails if the faithfulness score drops below our quality gate threshold.To execute the test suite in your CI pipeline, run: ```bash $ deepeval test run test_rag_evaluation.py ``` **Run Output:** ```bash Running test_rag_evaluation.py... ✓ test_rag_faithfulness (passed, score=1.0) ====================== 1 passed in 1.45s ====================== ```
🧠 Knowledge Check
Which metric in DeepEval evaluates whether the LLM's response is based entirely on the retrieved context without hallucinating external facts?