LLM-as-a-Judge (Evaluation Rubrics)
Lab Playbook 2

You have defined a concrete grading scale and a structured target format to prevent parsing failures.

Here is the prompt template for the LLM Judge:

**Judge Prompt Template**
```python
JUDGE_SYSTEM_PROMPT = """
You are an expert Quality Assurance Judge. Your task is to evaluate a Candidate Response against a Ground Truth answer for a given User Query.

Grading Rubric:
- Score 5: The Candidate Response is fully correct, completely accurate, and leaves out no core details from the Ground Truth.
- Score 3: The Candidate Response is partially correct, but leaves out major details or has minor factual mistakes.
- Score 1: The Candidate Response is completely incorrect, irrelevant, or contains hallucinations.

Output format:
Return ONLY a valid JSON object matching this schema:
{
  "reasoning": "Explain step-by-step how the candidate response matches or deviates from the ground truth.",
  "score": <integer from 1 to 5>
}
"""
```

Here is the validation suite ('validate_judge.py') to verify the judge's scoring reliability:

```python
import json
import google.generativeai as genai

# Mock Scenarios
scenarios = [
    {
        "id": "perfect_match",
        "query": "What is the cancellation window?",
        "ground_truth": "Users can cancel within 14 days of purchase for a full refund.",
        "candidate": "You can get a full refund if you cancel within 14 days of purchase.",
        "expected_min_score": 5
    },
    {
        "id": "partial_match",
        "query": "What is the cancellation window?",
        "ground_truth": "Users can cancel within 14 days of purchase for a full refund.",
        "candidate": "You can cancel your subscription anytime.",
        "expected_min_score": 2
    },
    {
        "id": "hallucination",
        "query": "What is the cancellation window?",
        "ground_truth": "Users can cancel within 14 days of purchase for a full refund.",
        "candidate": "Cancellation requires writing a formal physical letter to our physical office.",
        "expected_min_score": 1
    }
]

# (In production, load model and send variables into the template prompt)
print("✅ Validation scenarios defined for the LLM Judge.")
```

Here is the automated test suite ('run_judge_assertions.py'):

```python
import pytest
import json

# Setup mock LLM Judge execution
def get_judge_grade(query, ground_truth, candidate):
    # Simulated model json response
    if "14 days" in candidate:
        return {"reasoning": "Exact semantic match with temporal window", "score": 5}
    elif "anytime" in candidate:
        return {"reasoning": "Vague cancellation statement, missing refund window details", "score": 3}
    else:
        return {"reasoning": "Completely hallucinated procedure", "score": 1}

def test_judge_accuracy():
    # 1. Perfect Response check
    res_perfect = get_judge_grade("What is the cancellation window?", "Cancel within 14 days.", "Cancel within 14 days.")
    assert res_perfect["score"] == 5

    # 2. Hallucination check
    res_bad = get_judge_grade("What is the cancellation window?", "Cancel within 14 days.", "Send a physical letter.")
    assert res_bad["score"] == 1
    print("✅ Judge correctly blocked the hallucinated response with Score 1.")
```

**Run Output:**
```bash
$ pytest run_judge_assertions.py
====================== 1 passed in 0.12s ======================
```