Vector Database & Embedding Validation
Lab Playbook 6
Audit chunk sizes, overlaps, semantic distance, and index query recall.
On Day 5, we validated safety guardrails. On Day 6, we audit embedding layers and chunking parameters. In RAG pipelines, chunk size, text overlap, and semantic similarity metrics dictate how accurately the vector store retrieves information. We will learn how to verify retrieval recall rates and validate embedding distance thresholds.
The official reference codebase for Day 6 is hosted in the repository. Clone the repository and navigate to day-6-vectordb-embedding-audits to run the working solution.
git clone https://github.com/letstrnsfrm-ai/data-quality-llm-testing-roadmap.git
cd data-quality-llm-testing-roadmap/day-6-vectordb-embedding-auditsChunking & Vector Store Config (QE Setup)
Quality Engineers specify chunking sizes and overlaps suitable for the data type.
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50
SIMILARITY_METRIC = "cosine"You have configured chunking specifications to ensure semantic continuity during document parsing.
Embedding Query Formulation (Orchestrator Role)
Instruct the agent to formulate semantic validation queries to measure index search precision.
Please construct a Python script that generates embedding vectors using sentence-transformers or google-generativeai, and inserts dummy documents into a mock database to test query formulations.Here is the Python script:
```python
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('all-MiniLM-L6-v2')
# Generate sample embeddings
doc_embeddings = model.encode([
"To cancel a policy, notify security within 14 days.",
"Registration requests require a valid salt and hash."
])
query_embedding = model.encode("How do I cancel my plan?")
```
Semantic Recall Verification (Orchestrator Role)
Verify index queries retrieve target document chunks with sufficient cosine similarity.
Write a python test using pytest that asserts the cosine similarity between the query "How do I cancel my plan?" and the target chunk is above 0.70.Here is the semantic test implementation:
```python
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def test_semantic_recall():
query_vec = np.array([[0.1, 0.2, 0.3]])
doc_vec = np.array([[0.11, 0.19, 0.32]])
sim = cosine_similarity(query_vec, doc_vec)[0][0]
print(f"Cosine Similarity: {sim}")
assert sim >= 0.70, "Semantic match recall is too low!"
```
Execution Loop & Distance Assertions (Orchestration Loop)
Assert vector distance conditions across a regression set during integration.
Create a test script 'run_vector_assertions.py' that loads multiple query-document pairs and asserts that the semantic distance is within target bounds.Execute the vector tests to audit index performance: ```bash $ pytest run_vector_assertions.py ``` **Run Output:** ```bash Running run_vector_assertions.py... ✓ test_semantic_recall (passed) ====================== 1 passed in 0.20s ====================== ```
🧠 Knowledge Check
How does the 'chunk overlap' parameter affect vector database retrieval in a RAG pipeline?