PII, Safety, & Toxicity Guardrails
Lab Playbook 5
Evaluate output filtering policies and audit safety gates using Toxicity checks.
On Day 4, we modeled prompt injections. On Day 5, we look at safety guardrails. Deploying LLMs requires strict safety measures to mask Personally Identifiable Information (PII) and intercept toxic outputs. We will learn how to design rules that run on both inputs and outputs to ensure safety compliance.
The official reference codebase for Day 5 is hosted in the repository. Clone the repository and navigate to day-5-pii-safety-guardrails to run the working solution.
git clone https://github.com/letstrnsfrm-ai/data-quality-llm-testing-roadmap.git
cd data-quality-llm-testing-roadmap/day-5-pii-safety-guardrailsPII & Toxicity Policies (QE Setup)
Quality Engineers define policy parameters for toxic content classification and PII leakage.
CREDIT_CARD_PATTERN = r'\b(?:\d[ -]*?){13,16}\b'
SSN_PATTERN = r'\b\d{3}-\d{2}-\d{4}\b'
EMAIL_PATTERN = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'You have compiled baseline PII detection rules and toxicity thresholds for the integration pipelines.
Guardrail Rule Definition (Orchestrator Role)
Instruct the agent to construct an input/output sanitizer class enforcing the safety policies.
Please construct a Python helper class 'SafetyGuardrail' that:
1. Masks email addresses and credit cards in the input prompt with '[REDACTED]'.
2. Scans the generated output for toxic phrasing and replaces the output with a generic warning if toxicity is detected.Here is the Python class defining the guardrail:
```python
import re
class SafetyGuardrail:
@staticmethod
def mask_pii(text: str) -> str:
email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
masked = re.sub(email_pattern, "[REDACTED_EMAIL]", text)
return masked
@staticmethod
def check_toxicity(text: str) -> bool:
toxic_keywords = ["hate", "kill", "abuse", "steal"]
return any(word in text.lower() for word in toxic_keywords)
```
PII Masking & Safety Audits (Orchestrator Role)
Quality Engineers verify that PII is masked correctly and that the toxicity check rejects unsafe phrases.
Write a python script 'test_guardrails.py' that uses pytest to assert:
1. Inputs like "My email is test@domain.com" get masked to "My email is [REDACTED_EMAIL]".
2. Outputs containing toxic keywords trigger an assertion error or get successfully flagged by the check.Here is the test script ('test_guardrails.py'):
```python
import pytest
from guardrails import SafetyGuardrail
def test_pii_masking():
raw_input = "Hello, write to user.name@example.com."
expected = "Hello, write to [REDACTED_EMAIL]."
assert SafetyGuardrail.mask_pii(raw_input) == expected
def test_toxicity_filter():
unsafe_output = "I want to steal cash."
assert SafetyGuardrail.check_toxicity(unsafe_output) is True
```
Execution Loop & Safety Gates (Orchestration Loop)
Enforce the safety gates programmatically at the boundary lines of the execution run.
Create a pytest integration suite that passes all model calls through the SafetyGuardrail, ensuring a failed test blocks release if any output triggers the safety gates.Execute the test runner during build steps: ```bash $ pytest test_guardrails.py ``` **Run Output:** ```bash Running test_guardrails.py... ✓ test_pii_masking (passed) ✓ test_toxicity_filter (passed) ====================== 2 passed in 0.15s ====================== ```
🧠 Knowledge Check
At which stage of the LLM pipeline is it most effective to apply a Toxicity/PII guardrail?