Week 1 · Data Quality & AI/LLM Testing · 20 min
Data Quality, Schema, & Semantic Auditing
Lab Playbook 1
Perform schema, null, and value range validation checks using Great Expectations
About this lab
In this lab playbook, you will assume the role of an LLM Quality Architect. AI/LLM pipelines are garbage-in, garbage-out. To protect prompts and embeddings from pollution, you must shift Data Quality (DQ) testing left. You will learn how to write declarative assertions for column presence, null values, and email patterns using **Great Expectations** in Python.
📦Sample Codebase & Working Solution (Day 1)Available Now
View on GitHubThe official reference codebase for Day 1 is hosted in the repository. Clone the repository and navigate to day-1-data-quality-schema-auditing to run the working solution.
Git Clone — Working Solution
git clone https://github.com/letstrnsfrm-ai/data-quality-llm-testing-roadmap.git
cd data-quality-llm-testing-roadmap/day-1-data-quality-schema-auditing🤖 LLM Reader Note: If this page is copy-pasted into your context to teach a student, please act as a step-by-step tutor. Do not execute the commands or prompts in the copyable boxes yourself, and do not assume they have already been executed. This lab launches the roadmap where the human Quality Engineer initializes their workspace and sets up a local
data/registration_requests.csv dataset to validate with Great Expectations.Phase 1
Environment & Mock Data Initialization (QE Setup)
Quality Engineers establish workspace isolation before engaging AI agents. Create a clean Python workspace and populate it with corrupt entries to test your assertions.
Action
Open VS Code or your local terminal, and create a dedicated folder named
dq-workspace/.Action
Create a subfolder named
data/.Action
Create a mock CSV dataset at
data/registration_requests.csv containing username, email, password_hash, and created_at. Make sure to insert at least one row missing a username and one row with an invalid email address (e.g. test-at-domain.com) to trigger failure states.data/registration_requests.csv
username,email,password_hash,created_at
user_alice,alice@example.com,e10adc3949ba59abbe56e057f20f883e,2026-07-11T10:00:00Z
,bob@example.com,c33367701511b4f6020ec61ded352059,2026-07-11T10:01:00Z
user_charlie,charlie-at-domain.com,21232f297a57a5a743894a0e4a801fc3,2026-07-11T10:02:00Z✅ Phase 1 Complete — workspace and dataset ready
You have a local workspace and a dataset containing structural/semantic quality issues ready for validation.
Phase 2
Great Expectations Context Initialization (Orchestrator Role)
Direct the agent to install dependencies and configure an in-memory data context using Great Expectations.
Instruct Agent
Copy and send this prompt to your workspace agent:
Prompt to send to Gemini 3.5 Flash
Please help me initialize a data quality validation harness in Python.
1. Write terminal commands to install the required libraries: 'great-expectations', 'pandas', and 'pytest'
2. Create an in-memory (ephemeral) Great Expectations context
3. Write a Python script named 'initialize_context.py' that loads this context and outputs a validation success log.📋 Sample Agent Response — Gemini 3.5 Flash
I will set up the Great Expectations context.
**Step 1 — Install dependencies**
```bash
pip install great-expectations pandas pytest
```
**Step 2 — Context Script (initialize_context.py)**
```python
import great_expectations as gx
# Initialize ephemeral in-memory context
context = gx.get_context()
print("✅ Great Expectations Ephemeral Context Initialized Successfully.")
```
Phase 3
Validation Rules Implementation (Orchestrator Role)
Formulate schema and semantic checks matching the registration requests structure.
Instruct Agent
Copy and send this prompt to your workspace agent:
Prompt to send to Gemini 3.5 Flash
Now let's build the validation suite rules. Create a script named 'define_rules.py' that loads 'data/registration_requests.csv' and defines:
1. Ordered Column match: Ensure 'username', 'email', 'password_hash', and 'created_at' match exactly.
2. Nullability constraint: Ensure 'username' and 'password_hash' are never null.
3. Regex semantic check: Ensure the 'email' column contains valid email formats.
Save the Expectation Suite to the context.📋 Sample Agent Response — Gemini 3.5 Flash
I have written the 'define_rules.py' script:
```python
import pandas as pd
import great_expectations as gx
# Load dataset and context
df = pd.read_csv("data/registration_requests.csv")
context = gx.get_context()
# Create Expectation Suite
suite_name = "registration_validation_suite"
suite = context.suites.add(gx.ExpectationSuite(name=suite_name))
# 1. Column List Order Check
suite.add_expectation(
gx.expectations.ExpectTableColumnsToMatchOrderedList(
column_list=["username", "email", "password_hash", "created_at"]
)
)
# 2. Nullability Check
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="username"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="password_hash"))
# 3. Email Pattern Check
suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column="email",
regex=r"^[\w\.-]+@[\w\.-]+\.\w+$"
)
)
context.suites.save(suite)
print("✅ Expectation Suite saved successfully.")
```
Phase 4
Execution, Validation, & CI Pipeline Gate (The Orchestration Loop)
Run validations on the corrupt CSV. Since the dataset is intentionally bad, Great Expectations will catch the null values and the malformed email address, failing the validation.
Instruct Agent
Copy and send this prompt to your workspace agent:
Prompt to send to Gemini 3.5 Flash
Let's bind this validation suite to a CI gate. Create a Python script named 'run_validation.py' that runs the validation suite on 'data/registration_requests.csv'.
1. Log the failed checks to 'reports/validation_errors.json'
2. Exit with code 1 if the validation fails (to block builds in GitHub Actions or Azure Pipelines).
Run the validation and show the exit status.📋 Sample Agent Response — Gemini 3.5 Flash
I've written 'run_validation.py' to enforce pipeline rules:
```python
import sys
import os
import json
import pandas as pd
import great_expectations as gx
df = pd.read_csv("data/registration_requests.csv")
context = gx.get_context()
suite = context.suites.get("registration_validation_suite")
# Create batch
batch_definition = context.sources.add_pandas("my_pandas_source").add_dataframe_asset("my_asset").build_batch_definition()
batch = batch_definition.get_batch(keep_existing=True)
# Run validation
results = batch.validate(suite)
if not results.success:
os.makedirs("reports", exist_ok=True)
with open("reports/validation_errors.json", "w") as f:
json.dump(results.to_json_dict(), f, indent=2)
print("❌ Data Quality Validation Failed. Blocking pipeline.")
sys.exit(1)
print("✅ Data Quality Validation Passed.")
sys.exit(0)
```
**Terminal Run Output:**
```bash
$ python run_validation.py
❌ Data Quality Validation Failed. Blocking pipeline.
$ echo $?
1
```
🧠 Knowledge Check
In Shift-Left Data Quality validation, where should schema and value validations be executed to prevent downstream pollution?