DevOps Track · Day 3 State & Drift · 20 min
State & Drift Detection
DevOps Terraform Sprint — Day 3
Inspect state JSON schemas, simulate manual out-of-band drift, and enforce terraform plan -detailed-exitcode checks
About Today's 20-Minute Lab
Terraform relies on a state database (terraform.tfstate) to map real-world infrastructure to your HCL declarations.Infrastructure Drift occurs when resources are altered out-of-band (e.g. someone manually stops or reconfigures a container using Docker CLI). Today on Day 3, you will inspect state metadata, simulate manual drift, and master automated drift detection using terraform plan -detailed-exitcode.
🎬 Day 3 Video Walkthrough
⚡In a rush? Watch a quick knowledge bit in under 1 minute on today's lesson!
Watch Reel 📸📦Sample Codebase & Working Solution (Day 3)Available Now
View on GitHubThe official reference codebase for Day 3 is hosted in our private GitHub repository. Clone the repository and navigate to the day-3-state-drift directory to run the working solution.
Git Clone — Private Monorepo Solution
git clone https://github.com/letstrnsfrm-ai/devops-terraform-roadmap.git
cd devops-terraform-roadmap/day-3-state-drift1
Infrastructure Provisioning & State Inspection
GOAL
Apply a managed Nginx container and inspect the raw JSON state schema using CLI state subcommands.
MAIN.TF BLUEPRINT
Create `main.tf` for Day 3 lab:
HCL — main.tf
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
version = "~> 3.0.1"
}
}
}
provider "docker" {}
resource "docker_image" "nginx" {
name = "nginx:alpine"
keep_locally = false
}
resource "docker_container" "drift_demo" {
image = docker_image.nginx.image_id
name = "qe_drift_demo_container"
ports {
internal = 80
external = 8085
}
}STATE COMMANDS
Apply infrastructure and query state metadata:
Terminal — State Inspection
# Note for macOS Colima users: if socket error occurs, export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"
terraform init
terraform apply -auto-approve
# List all tracked resources in state
terraform state list
# Show detailed attributes of the container resource
terraform state show docker_container.drift_demo🏭Production Context & Enterprise Real-World Implementation
💡 Why We Are Doing This: State inspection gives platform engineers complete visibility into tracked resource IDs, IP addresses, and metadata without needing direct cloud console access.
🏢 Real-World Production Usecase: Production platform teams at Uber or Datadog query Terraform state via CLI tools (`terraform state show`) to build automated asset inventories and verify IP range allocations.
⚙️ How Implemented in Production: In production, state files are stored in remote encrypted backends (like AWS S3 with KMS encryption + DynamoDB locking). Direct edits to `.tfstate` files are strictly blocked via IAM permission policies.
2
Out-of-Band Manual Drift Simulation
GOAL
Bypass Terraform and modify the running container directly via Docker CLI to introduce real infrastructure drift.
OUT-OF-BAND EDIT
Stop and delete the running container manually without updating Terraform:
Terminal — Simulate Manual Drift
# 1. Stop and remove the container behind Terraform's back
docker stop qe_drift_demo_container
docker rm qe_drift_demo_container
# 2. Verify container is missing from Docker CLI
docker ps -a | grep qe_drift_demo_containerEXPECTED OUTPUT
Out-of-Band State
The container is deleted in Docker, but
terraform.tfstate still records it as running. This discrepancy is Infrastructure Drift.🏭Production Context & Enterprise Real-World Implementation
💡 Why We Are Doing This: Infrastructure drift is a top cause of cloud outages. Out-of-band changes (e.g. an engineer manually modifying a security group or terminating an EC2 instance via UI) cause real infrastructure to deviate from code.
🏢 Real-World Production Usecase: Fintech infrastructure teams (like Stripe or Square) strictly enforce zero-drift policies. Unrecorded manual security group edits violate SOC2 compliance and expose internal databases to the public internet.
⚙️ How Implemented in Production: Production AWS environments enforce Read-Only access on cloud console UIs for human engineers. All infrastructure changes must pass through Git commits and automated CI pipelines.
3
Drift Audit with -detailed-exitcode
GOAL
Use
terraform plan -detailed-exitcode in automated scripts to detect drift without applying changes.DETAILED EXIT CODES
Execute plan with detailed exit code flags:
Terminal — Execute Detailed Exitcode Check
# Exit codes: 0 = Succeeded (no changes), 1 = Error, 2 = Succeeded (changes/drift present)
# In CI shell scripts (set -e), capture exit code without failing pipeline:
terraform plan -detailed-exitcode || EXIT_CODE=$?
echo "Recorded Exit Code: $EXIT_CODE"EXPECTED OUTPUT
Drift Detected
Exit Code
2 is returned! Terraform detects that docker_container.drift_demo must be re-created to restore the desired declarative state.🏭Production Context & Enterprise Real-World Implementation
💡 Why We Are Doing This: Automated scripts require exact status exit codes (0 = no drift, 2 = drift present) to fail CI pipelines or send PagerDuty alerts without relying on text parsing.
🏢 Real-World Production Usecase: Enterprise platforms run automated hourly drift detection cron jobs across thousands of cloud resources. If exit code `2` is returned, a Slack notification or Jira ticket is automatically dispatched.
⚙️ How Implemented in Production: Tools like Driftctl or Spacelift execute `terraform plan -detailed-exitcode` on scheduled crons, alerting platform engineers when production resources drift from Git main.
4
Automated Reconciliation & Cleanup
GOAL
Reconcile infrastructure back to the desired blueprint and destroy resources cleanly.
RECONCILE WORKFLOW
Run apply to automatically recreate the missing container:
Terminal — Reconcile & Destroy
# Re-create missing container to restore desired state
terraform apply -auto-approve
# Verify HTTP response on port 8085
curl -I http://localhost:8085
# Tear down lab resources
terraform destroy -auto-approveEXPECTED OUTPUT
Terraform self-heals by re-provisioning the container, returning
Verification Success
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Terraform self-heals by re-provisioning the container, returning
HTTP/1.1 200 OK.🏭Production Context & Enterprise Real-World Implementation
💡 Why We Are Doing This: Declarative IaC allows infrastructure self-healing. Running `terraform apply` automatically replaces deleted or corrupt instances to restore 100% operational availability.
🏢 Real-World Production Usecase: High-availability cloud platforms (like Netflix or Airbnb) rely on automated self-healing to replace terminated microservice instances during cloud zone outages.
⚙️ How Implemented in Production: GitOps controllers (like ArgoCD or Terraform Controller) monitor live clusters continuously, automatically running `apply` to reconcile state whenever drift is detected.