Business Case: LLM-Driven Exploratory Testing Framework
SDLC Phase: Planning & Refinement
Executive Summary
Implementing an LLM-powered CLI utility to automate requirement analysis from Azure DevOps and execute exploratory testing via Playwright. Scaling this across the entire Quality Engineering team will significantly reduce manual overhead, accelerate delivery, and yield a high Return on Investment (ROI).
Architectural Decisions & Technical Trade-offs
To ensure predictability, cost-efficiency, and seamless integration into existing pipelines, specific choices were made regarding data fetching and test execution tooling.
1. Requirements Ingestion: Custom Node.js CLI vs. Alternatives
The project utilises a custom Node.js CLI built with the official Microsoft API to extract Azure DevOps (ADO) work items as local Markdown files.
| Approach | Pros | Cons |
|---|---|---|
| Custom Node.js CLI (Chosen) | Complete control over data formatting; aggregates descriptions, acceptance criteria, child links, and comments into a single LLM-optimised Markdown file in one run; zero token cost for ingestion. | Requires initial development and minor long-term maintenance. |
| ADO MCP Server | Native integration with LLM clients out of the box. | High token latency and cost; requires the LLM to make multiple chat round-trips to discover and crawl links or child items; inconsistent formatting. |
| Official ADO CLI | No custom authentication code required; maintained by Microsoft. | Outputs raw JSON or text tables; requires heavy shell scripting wrappers to parse nested data and structure it into clean Markdown. |
2. Test Execution: Playwright CLI vs. Playwright MCP Server
The framework instructs the LLM to interact with the application using the Playwright CLI rather than deploying a continuous Model Context Protocol (MCP) browser loop.
| Approach | Pros | Cons |
|---|---|---|
| Playwright CLI (Chosen) | Highly deterministic; execution scripts can be audited, reused, and integrated directly into standard CI/CD pipelines; predictable token spend. | Requires the LLM to generate the execution plan or script prior to running, rather than reacting live to every click. |
| Playwright MCP Server | Allows the LLM to dynamically react to the browser state step-by-step, making it highly autonomous during unguided exploratory testing. | Extremely high token consumption due to constant DOM/screenshot transfers; prone to getting stuck in infinite navigation loops; difficult to replicate deterministically. |
Technical Recommendation
The combination of a Custom Node.js CLI and the Playwright CLI is recommended. This hybrid approach guarantees that data extraction is perfectly structured for the LLM without wasting API tokens, while the Playwright CLI keeps test execution stable, auditable, and ready for future pipeline integration.
Tangible Example: LLM-Optimised ADO Work Item Markdown
The custom Node.js CLI transforms complex Azure DevOps JSON responses into a single, flattened Markdown file that is easy for LLMs to parse and reason about.
File: requirements/ADO-12345-search-logic.md
# [User Story 12345] As a user, I want to search for policies by registration number
## Description
The search bar should accept alpha-numeric strings and return matching policy records.
## Acceptance Criteria
- [ ] Must return 0 results if no match is found.
- [ ] Must handle special characters gracefully.
- [ ] Must be case-insensitive.
## Technical Context (Linked Items)
- **Parent:** Feature 9876 - Policy Management UI
- **Relation:** Blocks Bug 54321 - Search Timeout in Staging
## Engineer Comments
> @jules: "We should ensure the debounce is set to at least 300ms to avoid overloading the API."
ROI & Commercial Benefits
- 70% Reduction in Test Design Time: Automating ADO context extraction and test case generation frees engineers to focus on complex edge cases rather than boilerplate documentation.
- Accelerated Time-to-Market: Faster test generation and automated Playwright execution remove the testing bottleneck, allowing for quicker, more predictable release cycles.
- Decreased Defect Leakage: AI-driven, risk-based analysis identifies critical business paths early, reducing the high cost of fixing bugs in production.
- Standardised Quality: Ensures consistent, high-quality test coverage and documentation across the entire engineering department, regardless of an individual engineer's bandwidth.
- Low Infrastructure Cost: Utilising a docs-as-code approach with local Markdown files and existing ADO/Playwright infrastructure ensures the only new operational cost is LLM API token consumption. This expense is heavily outweighed by the engineering hours saved.
Conclusion
Adopting this agentic workflow across all quality engineers transforms the testing lifecycle from a manual constraint into an automated, scalable asset. The minimal initial development investment will deliver immediate operational efficiency and substantial long-term cost savings.