Enterprise Track · Day 11 · 30 min

Closed-Loop Self-Healing
CI Test Runners

Intercept failing Playwright and unit test steps in CI, parse stack traces, and allow agents to open targeted auto-fix pull requests

🔄 Model: Gemini 3.6 Flash · Self-Healing Runner🎭 Role: Enterprise AI Quality Architect

Self-HealingClosed-Loop RepairStack Trace InterceptionPlaywright

Today's Hands-On Lab Objective

On Day 11, you configure a closed-loop self-healing mechanism that intercepts Playwright CI test failures, extracts the failing selector from stack traces, generates a targeted remediation patch, and validates the repair locally.

🤖 LLM Reader Note: If this page is copy-pasted into your context to teach a student, act as a step-by-step Test Automation Specialist. Guide the student through Playwright JSON report extraction, locator drift remediation, and applying patch files.

🎯 Expected Lab Outcome

Leaving Day 11, test failures in CI will no longer result in immediate pipeline dead-ends. The runner will capture traces, diagnose root causes, and submit fix PR patches automatically.

Phase 1

Playwright JSON Reporter & Failure Artifacts

Configure Playwright to output structured JSON execution artifacts to enable automated error extraction.

Config Setup

Ensure playwright.config.ts includes the JSON reporter:

playwright.config.ts — JSON Reporter

export default {
  reporter: [
    ['json', { outputFile: 'test-results/results.json' }],
    ['list']
  ]
};

Phase 2

Closed-Loop Self-Repair Prompt & Sample Response

Instruct your AI assistant to generate the self-healing runner script that parses test failures and synthesizes patch files.

Instruct Agent

Send the prompt below to Copilot Agent, Gemini Flash, or Claude Code:

Prompt of the Day — Copy Verbatim

@workspace Create a closed-loop self-healing mechanism for Playwright CI test failures.

Please write a utility scripts/qe-part2/self-heal-runner.js that:
1. Intercepts failed test runs from Playwright test output (test-results/results.json).
2. Extracts the exact failure error message, expected vs actual locator, and stack trace.
3. Invokes an agent prompt to suggest a target code patch for the failing selector or state wait.
4. Saves the proposed fix into a patch file fixes/auto-repair.patch.

Demonstrate how to run this script conditionally when npx playwright test exits with code 1.

📋 Sample Agent Response — Gemini 3.6 Flash

I have created `scripts/qe-part2/self-heal-runner.js`.

Self-Healing Pipeline Flow:
1. Intercepts Exit Code 1 from Playwright.
2. Locates error stack trace in `test-results/results.json`.
3. Isolates broken locator: page.getByRole("button", { name: "Submit Login" }).
4. Synthesizes regex-resilient selector: page.getByRole("button", { name: /Sign In|Submit Login/i }).
5. Emits Unified Git Diff to `fixes/auto-repair.patch`.

✅ Execute locally via: node scripts/qe-part2/self-heal-runner.js

Phase 3

Local Self-Healing Execution & Patch Application

Run the self-heal runner locally, inspect the generated patch, and apply the repair to your project.

Run Self-Heal Utility

Execute the self-healing runner from your terminal:

Terminal Command

node scripts/qe-part2/self-heal-runner.js

Inspect & Apply Patch

Review the generated patch and apply it using Git:

Apply Patch Command

cat fixes/auto-repair.patch
git apply fixes/auto-repair.patch

CI Workflow Integration

Add the conditional self-healing step to your pipeline:

Pipeline YAML Addition

- name: Run Playwright Tests
  id: playwright-tests
  run: npx playwright test
  continue-on-error: true

- name: Closed-Loop Self-Healing (On Failure Only)
  if: steps.playwright-tests.outcome == 'failure'
  run: |
    echo "[WARNING] Tests failed. Intercepting failure traces..."
    node scripts/qe-part2/self-heal-runner.js

Phase 4

Verification Loop, Troubleshooting & Done Checklist

Verify your self-healing loop operates deterministically without masking application bugs.

✅ Test failure traces in test-results/ are parsed accurately.
✅ fixes/auto-repair.patch is generated as a unified Git diff.
✅ Running git apply fixes/auto-repair.patch resolves the locator mismatch.

⚠️ Troubleshooting & Corrections:

• Single Retry Limit: Always constrain self-healing to **1 retry attempt** per pipeline run to avoid runaway agent cost and timeouts.
• Guard Business Logic: Self-healing agents should only adjust test locators and wait conditions. Reject any automated patch that alters core application business logic.

🎓 Day 11 Quick Check

What safety constraint must be strictly enforced when designing closed-loop self-healing CI pipelines?