Chaos Engineering Track · Day 1 Foundation · 20 min

Foundations of Cloud Chaos & Fault Injection
Chaos & SRE Sprint — Day 1

Establish a formal Steady-State Hypothesis, author a Kubernetes Chaos Mesh NetworkChaos delay manifest, and empirically verify SLA compliance under injected 45ms + 10ms jitter latency.

✨ Agent: Claude 3.7 Sonnet / Gemini 3.5 · 200K context🔥 Chaos & Site Reliability Quality Architect

Chaos MeshNetworkChaosSteady-State HypothesisFault InjectionSLI/SLO

About this lab

In this lab playbook, you will assume the role of a Chaos & Site Reliability Quality Architect. Before running destructive faults against production microservices, every Chaos experiment must begin with a measurable Steady-State Hypothesis on our 3-tier target system:api-gateway → policy-engine → ledger-db.

📦Sample Codebase & Working Solution (Day 1)Available Now

View on GitHub

The official reference codebase for Day 1 is hosted in the repository. Clone the repository and navigate to day-1-fault-injection to run the working solution.

Git Clone — Working Solution

git clone https://github.com/letstrnsfrm-ai/chaos-engineering-roadmap.git
cd chaos-engineering-roadmap/day-1-fault-injection

🏭Production Context & Enterprise Implementation

💡 Why We Are Doing This: Microservice architectures suffer from hidden timeout cascades when downstream RPC calls experience mild network jitter. Without proactive latency injection, mismatched client/server timeouts cause retry storms in production.

🏢 Real-World Production Usecase: Validating that a 45ms + 10ms jitter network delay between api-gateway and policy-engine does not breach the 200ms P95 checkout SLO or trigger premature retry amplification.

⚙️ How Implemented in Production: Kubernetes Chaos Mesh NetworkChaos CRD targeting app=policy-engine pods paired with automated SLI baseline assertions.

🤖 LLM Reader Note: If this page is copy-pasted into your context, act as an SRE Chaos Engineering tutor. Guide the human engineer through defining a steady-state hypothesis, authoring a Chaos Mesh NetworkChaos manifest, and verifying latency resilience.

Phase 1

Define Steady-State Hypothesis & Chaos Mesh CRD

Clone the reference repository and inspect the Steady-State Hypothesis contract and Kubernetes Chaos Mesh NetworkChaos manifest.

Step 1.1: Clone & Inspect Baseline

Terminal — Setup Day 1 Lab

git clone https://github.com/letstrnsfrm-ai/chaos-engineering-roadmap.git
cd chaos-engineering-roadmap/day-1-fault-injection
cat sli-baseline.json

Step 1.2: Chaos Mesh NetworkChaos Manifest

network-delay-chaos.yaml

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: policy-engine-latency-injection
  namespace: checkout-mesh
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - checkout-mesh
    labelSelectors:
      app: policy-engine
  delay:
    latency: "45ms"
    correlation: "50"
    jitter: "10ms"
  duration: "2m"

Phase 2

Execute Fault Injection & Assert SLO Compliance

Run the empirical latency fault-injection harness against the Resilient Checkout Mesh and verify all checkout transactions remain within the 200ms P95 SLO.

Step 2.1: Run Empirical Latency Drill

Terminal — Execute Day 1 Verification

cd ..
npm run test:day1

Day 1 Key Learnings

Defined a formal Steady-State Hypothesis (P95 <= 200ms, Success Rate >= 99.5%) prior to fault injection.
Authored a Chaos Mesh NetworkChaos delay CRD with 45ms latency and 10ms jitter.
Verified empirically that checkout transactions absorb downstream latency without breaching SLO thresholds.

🧠 Knowledge Check

Why must every Chaos Engineering experiment define a quantitative Steady-State Hypothesis before injecting faults?

Next: Day 2 — Network Partitions & Packet Loss →