Chaos Engineering Track · Day 4 · 20 min

Database Deadlocks & Storage Exhaustion
Chaos & SRE Sprint — Day 4

Inject IOChaos disk exhaustion and database deadlocks into ledger-db, and verify the full Circuit Breaker state machine (CLOSED → OPEN → HALF_OPEN → CLOSED).

✨ Agent: Claude 3.7 Sonnet / Gemini 3.5 · 200K context🔥 Chaos & Site Reliability Quality Architect

IOChaosCircuit BreakerDatabase DeadlockGraceful DegradationENOSPC

About this lab

In this lab playbook, you will inject a storage deadlock / disk exhaustion fault (errno: 28 ENOSPC) into ledger-db and verify that the upstream Circuit Breaker trips from CLOSED toOPEN after 3 consecutive failures, returns fast degraded queued responses (DEGRADED_QUEUED), probes HALF_OPEN after cooldown, and recovers to CLOSED.

📦Sample Codebase & Working Solution (Day 4)Available Now

View on GitHub

The official reference codebase for Day 4 is hosted in the repository. Clone the repository and navigate to day-4-database-deadlocks to run the working solution.

Git Clone — Working Solution

git clone https://github.com/letstrnsfrm-ai/chaos-engineering-roadmap.git
cd chaos-engineering-roadmap/day-4-database-deadlocks

🏭Production Context & Enterprise Implementation

💡 Why We Are Doing This: When a database experiences row lock contention or disk I/O saturation, upstream worker threads block until connection pools exhaust, taking down the entire API gateway.

🏢 Real-World Production Usecase: Tripping an upstream Circuit Breaker after 3 consecutive ledger-db timeouts and queuing orders asynchronously (DEGRADED_QUEUED) instead of failing checkouts.

⚙️ How Implemented in Production: Chaos Mesh IOChaos (errno: 28) paired with a state-machine Circuit Breaker drill verifying CLOSED -> OPEN -> HALF_OPEN -> CLOSED.

🤖 LLM Reader Note: If this page is copy-pasted into your context, act as an SRE Chaos tutor. Guide the engineer through IOChaos fault injection and Circuit Breaker state-machine verification.

Phase 1

Inspect IOChaos Disk Exhaustion Manifest

Review the Chaos Mesh IOChaos manifest injecting errno: 28 (No space left on device) into PostgreSQL data volumes.

Step 1.1: IOChaos Manifest

io-fault-chaos.yaml

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: ledger-db-disk-deadlock
  namespace: checkout-mesh
spec:
  action: fault
  mode: one
  selector:
    namespaces:
      - checkout-mesh
    labelSelectors:
      app: ledger-db
  volumePath: /var/lib/postgresql/data
  path: "/var/lib/postgresql/data/**/*"
  errno: 28 # ENOSPC / Disk I/O exhaustion
  percent: 100
  duration: "60s"

Phase 2

Verify Circuit Breaker Trip, Fast Fallback & Recovery

Execute the circuit-breaker drill to verify all four state transitions (CLOSED → OPEN → HALF_OPEN → CLOSED).

Step 2.1: Run Circuit Breaker Drill

Terminal — Execute Day 4 Verification

npm run test:day4

Day 4 Key Learnings

Simulated database deadlock and disk I/O exhaustion (errno 28 ENOSPC).
Verified Circuit Breaker tripped from CLOSED to OPEN after 3 consecutive failures.
Confirmed fast fallback (200 DEGRADED_QUEUED) prevented upstream thread pool exhaustion.
Verified automatic HALF_OPEN probe and recovery to CLOSED once the DB lock cleared.

🧠 Knowledge Check

What is the role of the HALF_OPEN state in a Circuit Breaker pattern?

← Previous: Day 3 Pod TerminationsNext: Day 5 — Multi-Agent Chaos Orchestration →