Chaos Engineering Track · Day 3 · 20 min

Pod Terminations & Kubernetes Node Evictions
Chaos & SRE Sprint — Day 3

Trigger ungraceful SIGKILL pod terminations via PodChaos, verify zero-downtime request balancing across surviving replicas, and confirm ReplicaSet self-healing.

✨ Agent: Claude 3.7 Sonnet / Gemini 3.5 · 200K context🔥 Chaos & Site Reliability Quality Architect

PodChaosSIGKILLReplicaSet Self-HealingZero-DowntimePodDisruptionBudget

About this lab

In this lab playbook, you will execute an immediate gracePeriod: 0 pod termination onpolicy-pod-a using Chaos Mesh PodChaos, verify that surviving replicas (policy-pod-b, policy-pod-c) absorb 100% of inflight requests with zero 5xx errors, and assert automatic ReplicaSet replacement.

📦Sample Codebase & Working Solution (Day 3)Available Now

View on GitHub

The official reference codebase for Day 3 is hosted in the repository. Clone the repository and navigate to day-3-pod-terminations to run the working solution.

Git Clone — Working Solution

git clone https://github.com/letstrnsfrm-ai/chaos-engineering-roadmap.git
cd chaos-engineering-roadmap/day-3-pod-terminations

🏭Production Context & Enterprise Implementation

💡 Why We Are Doing This: Spot instance reclamations, OOMKills, and Kubernetes node drains evict pods without warning. If readiness probes or PodDisruptionBudgets are misconfigured, users receive HTTP 502/503 errors during pod churn.

🏢 Real-World Production Usecase: Validating zero-downtime checkout processing during an ungraceful SIGKILL eviction of an active policy-engine replica.

⚙️ How Implemented in Production: Chaos Mesh PodChaos (action: pod-kill, mode: one, gracePeriod: 0) with concurrent synthetic traffic assertions.

🤖 LLM Reader Note: If this page is copy-pasted into your context, act as an SRE Chaos tutor. Guide the engineer through testing ungraceful pod kills and ReplicaSet self-healing.

Phase 1

Inspect PodChaos SIGKILL Specification

Review the PodChaos manifest configured with gracePeriod: 0 to simulate a sudden kernel OOMKill or spot node termination.

Step 1.1: PodChaos YAML

pod-kill-chaos.yaml

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: policy-engine-random-pod-kill
  namespace: checkout-mesh
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - checkout-mesh
    labelSelectors:
      app: policy-engine
  gracePeriod: 0

Phase 2

Execute Pod Kill Drill & Verify Self-Healing

Run the pod eviction drill and confirm both zero 5xx errors during eviction and automatic restoration of the 3-replica pool.

Step 2.1: Run Pod Kill Drill

Terminal — Execute Day 3 Verification

npm run test:day3

Day 3 Key Learnings

Triggered ungraceful PodChaos pod-kill (gracePeriod: 0) against an active service replica.
Confirmed surviving replicas handled 100% of concurrent requests without HTTP 5xx errors.
Verified ReplicaSet controller self-healing restored the replica count back to 3.

🧠 Knowledge Check

Which Kubernetes resource prevents voluntary node drains from evicting all replicas of a critical stateless service simultaneously?

← Previous: Day 2 Network PartitionsNext: Day 4 — Database Deadlocks & Circuit Breakers →