Chaos Engineering Track · Day 5 · 20 min

Multi-Agent Chaos Orchestration
Chaos & SRE Sprint — Day 5

Coordinate an adversarial Chaos Injector Agent with an autonomous SRE Guardian Agent that enforces live SLI abort gates and triggers immediate rollback upon SLO breach.

✨ Agent: Claude 3.7 Sonnet / Gemini 3.5 · 200K context🔥 Chaos & Site Reliability Quality Architect

Multi-Agent ChaosSLI Abort GateAutomated RollbackError BudgetGame Day

About this lab

In this lab playbook, you will orchestrate a dual-agent chaos campaign: the Chaos Injector Agentescalates fault severity across 3 steps while the SRE Guardian Agent continuously monitors live SLI telemetry and automatically aborts and rolls back the experiment the instant latency exceeds 150msor HTTP 5xx errors occur.

📦Sample Codebase & Working Solution (Day 5)Available Now

View on GitHub

The official reference codebase for Day 5 is hosted in the repository. Clone the repository and navigate to day-5-multi-agent-chaos to run the working solution.

Git Clone — Working Solution

git clone https://github.com/letstrnsfrm-ai/chaos-engineering-roadmap.git
cd chaos-engineering-roadmap/day-5-multi-agent-chaos

🏭Production Context & Enterprise Implementation

💡 Why We Are Doing This: Unbounded chaos experiments can cause self-inflicted production outages. Every enterprise chaos runner must pair fault injection with an automated SLI abort circuit-breaker (Emergency Stop).

🏢 Real-World Production Usecase: Allowing an autonomous Chaos Agent to probe progressive fault combinations while guaranteeing immediate auto-rollback if checkout latency exceeds 150ms.

⚙️ How Implemented in Production: Declarative abort-policy.json guardrails enforced by a closed-loop SRE Guardian Agent.

🤖 LLM Reader Note: If this page is copy-pasted into your context, act as an SRE Chaos tutor. Guide the engineer through configuring automated SLI abort policies for multi-agent chaos campaigns.

Phase 1

Configure Automated SLI Abort Policy

Inspect abort-policy.json defining the 3-step progressive attack campaign and the hard SLI abort guardrails (maxAllowedLatencyMs: 150).

Step 1.1: Abort Policy Contract

abort-policy.json

{
  "experimentName": "autonomous-multi-agent-chaos-campaign",
  "sliGuardrails": {
    "maxAllowedLatencyMs": 150,
    "maxConsecutiveErrors": 2,
    "autoRollbackOnBreach": true
  },
  "attackSequence": [
    { "step": 1, "name": "Mild Latency Probe (30ms)", "config": { "latencyMs": 30, "dbDeadlockActive": false } },
    { "step": 2, "name": "Regional Partition Failover", "config": { "latencyMs": 20, "partitionPrimaryRegion": true } },
    { "step": 3, "name": "Severe Latency + Deadlock Breach (180ms)", "config": { "latencyMs": 180, "dbDeadlockActive": true } }
  ]
}

Phase 2

Run Dual-Agent Campaign & Verify Emergency Rollback

Execute the multi-agent orchestrator and verify that Steps 1 and 2 pass, Step 3 trips the SLI abort guardrail, and the SRE Guardian immediately restores steady state.

Step 2.1: Execute Multi-Agent Drill

Terminal — Execute Day 5 Verification

npm run test:day5

Day 5 Key Learnings

Configured progressive multi-step fault escalation with quantitative SLI abort guardrails.
Verified the SRE Guardian Agent halted the experiment immediately when Step 3 breached the 150ms latency ceiling.
Confirmed automated rollback restored 200 COMMITTED steady state without manual intervention.

🧠 Knowledge Check

Why is an automated SLI Abort Gate (Emergency Stop) mandatory when running autonomous chaos agents in production or pre-production environments?

← Previous: Day 4 Database DeadlocksNext: Day 6 — Automated Post-Mortem & Blast Radius →