ModelOps CI/CD Validation Gates
Lab Playbook 7
Establish deployment regressions gates based on evaluation score metrics.
On Day 6, we validated vector store indexes. On Day 7, we build automated deployment validation gates. ModelOps requires regression gates to ensure new prompt updates or base model upgrades do not compromise response quality. We will learn how to build test automation workflows that reject builds if accuracy metrics degrade.
The official reference codebase for Day 7 is hosted in the repository. Clone the repository and navigate to day-7-modelops-cicd-gates to run the working solution.
git clone https://github.com/letstrnsfrm-ai/data-quality-llm-testing-roadmap.git
cd data-quality-llm-testing-roadmap/day-7-modelops-cicd-gatesCI/CD Gate Integration (QE Setup)
Quality Engineers configure target thresholds and gate definitions in the project CI runner configuration.
FAITHFULNESS_THRESHOLD = 0.80
ANSWER_RELEVANCE_THRESHOLD = 0.85
STRICT_SAFETY_GATE = trueYou have codified minimum quality metrics to serve as exit criteria for the CI release candidate builds.
Threshold & Validation Modeling (Orchestrator Role)
Instruct the agent to write a python test wrapper parsing evaluation logs and enforcing exit codes.
Please construct a Python helper script 'verify_modelops_gate.py' that loads a JSON test report containing scores, checks them against our thresholds, and exits with 1 if any score fails.Here is the Python verification script:
```python
import sys
import json
def verify_gate(report_path: str):
with open(report_path) as f:
data = json.load(f)
# Assert faithfulness scores are above threshold
for run in data.get("runs", []):
if run["faithfulness"] < 0.80:
print(f"❌ Regression detected on run: {run['id']}")
sys.exit(1)
print("✅ All ModelOps gates successfully passed.")
sys.exit(0)
```
Regression Gate Verification (Orchestrator Role)
Verify that a regression correctly triggers a build failure and logs descriptive errors.
Write a python test suite using pytest to simulate a validation failure scenario.
The test should load a report mock with a faithfulness score of 0.50 and assert that verify_gate raises a SystemExit exception with status code 1.Here is the simulated regression verification test:
```python
import pytest
def test_regression_gate_failure():
# Simulated failing metrics
failing_runs = [{"id": "run_01", "faithfulness": 0.50}]
with pytest.raises(SystemExit) as excinfo:
# Code simulating gate validation check
if any(run["faithfulness"] < 0.80 for run in failing_runs):
import sys
sys.exit(1)
assert excinfo.value.code == 1
```
Execution Loop & Deployment Assertions (Orchestration Loop)
Integrate the gate check script into the repository build flow to secure production releases.
Show the CI configuration block to execute the regression gate script after running deepeval validation metrics.Here is the step description for GitHub Actions: ```yaml - name: Run RAG Evaluation Tests run: deepeval test run test_rag_evaluation.py --junit-xml=report.xml - name: Execute ModelOps Regression Gate run: python verify_modelops_gate.py --report=report.json ```
🧠 Knowledge Check
What is the primary purpose of a ModelOps CI/CD validation gate?