Systems fall over. Not sometimes—always, eventually. Disks go bad. Configs drift in ways nobody notices until it’s 2 a.m. and the pager won’t stop buzzing. And when the pressure lands, human error doesn’t just creep in; it barges through the door. What separates a brief wobble from a full-blown outage is often just one thing: a recovery checklist written back when the world was quiet and the coffee was still warm. This piece walks through building one that’s specific, operational, and ready for the moment you’d rather not think about.

Why Pre-Incident Documentation Matters
When an outage hits, your brain works against you. Cognitive load spikes, the prefrontal cortex narrows, and tunnel vision takes over. You stop seeing options and start grasping at the first thing that looks like a fix. A checklist works like an external hard drive for your memory—it holds the steps so you can stay present and think instead of flail. At the Gray Haven Lab, we call checklists operational prosthetics. They carry the load so the operator doesn’t have to.
But don’t confuse them with runbooks. A runbook assumes a known problem and a known path. A recovery checklist lives in the fog—when the root cause is unclear, the blast radius is still growing, and the first real job is simply to stop the bleeding. Writing the Recovery Checklist Before You Need It digs into that distinction a bit more. Here, we stay focused on the structure you can build today.
Defining Scope and Triggers
A checklist with no boundaries turns into a novel nobody will read at 3 a.m. Pin down exactly what it covers. One application? A whole region? A database cluster? Be blunt about it. Then nail the triggers—which alerts, metrics, or human reports should make someone grab this document and not some other one.
- PagerDuty alert for service latency above 5000ms sustained for 5 minutes.
- Monitoring dashboard shows disk utilization over 95% on primary nodes.
- Customer reports of complete checkout failure on the web storefront.
Stick these triggers right at the top. The first thing a responder should do is confirm they’re holding the right piece of paper—or the right terminal window.

Anatomy of a Recovery Checklist
A checklist that works has five sections. Each one does a specific job for your brain.
1. Immediate Safety Checks
Before you touch anything, make sure nobody’s in physical danger and that you can contain the mess. In digital systems, that usually means isolating compromised segments. A few sample items:
- Confirm no electrical hazard at rack location.
- Verify that monitoring data is still flowing to an off-site aggregator.
- Isolate affected VLAN if breach indicators are present.
2. Triage and Assessment
Collect information first. Don’t change anything yet. The urge to “just restart it” is strong, but premature tinkering can shred forensic evidence or make things worse. Typical checklist lines:
- Run
dmesgor equivalent for recent kernel panics. - Check systemd journal for the last 15 minutes of logs.
- Capture current network connection states with
ss -tulpn. - Snapshot relevant metrics: CPU, memory, I/O wait, disk queue depth.
3. Stabilization Steps
Once you know roughly what you’re dealing with, move toward restoring basic function—even if it’s limping. Order the steps by least destructive first. For a database cluster, the sequence might run:
- Restart read replicas one at a time, waiting for sync check after each.
- If replicas are healthy, perform a controlled failover to a replica.
- If failover fails, initiate restore from the most recent verified backup.
Each step should name an expected outcome and a rollback instruction for when the outcome doesn’t show up.
4. Communication Cadence
During incidents, silence eats trust. Stakeholders need updates even when the update is “we still don’t know.” Spell out who communicates, through which channels, and how often. For example:
- Engineering lead posts to #incident-response Slack channel every 30 minutes.
- Customer support updates the status page at 60-minute intervals.
- If the incident exceeds 2 hours, the on-call manager sends an email to executive sponsors.
5. Post-Recovery Verification
A green light on a dashboard doesn’t mean the patient is healthy. List the checks that confirm real recovery:
- Run synthetic transactions against the production endpoint.
- Verify replication lag is under 2 seconds.
- Check that monitoring alerts have returned to green for 10 consecutive minutes.
- Confirm that backup jobs are scheduled and not paused.

Writing Style and Formatting
Checklists die when they get wordy or vague. A few rules keep them alive:
- Use the imperative mood: “Check disk space” not “You should check disk space.”
- Keep line length under 80 characters if the checklist will be viewed in a terminal.
- Use a monospaced font for commands and file paths.
- Number steps only when order matters; otherwise, stick with bullets.
- Include blank checkboxes (
[ ]) for manual verification steps.
Store the checklist where it’s reachable even when primary infra is toast. At Gray Haven Lab, critical checklists live in a private Git repository mirrored to a USB drive inside the physical ops binder. More than one late-night rescue has started with a printed copy taped inside the server cabinet door.
Testing the Checklist
An untested checklist is a wish list. Run a tabletop exercise at least every quarter. Pull the on-call team together, toss them a scenario, and walk the checklist without laying a finger on production. Note every rough edge: missing dependencies, stale commands, wording that made someone pause. Fix the checklist right after the exercise. If your team practices Writing the Recovery Checklist Before You Need It, they’ll already know the document is a living thing, not a one-and-done artifact.
Common Pitfalls
Over-prescribing. A checklist that tries to cover every edge case becomes a swamp. Leave room for human judgment. Write steps like “If primary database is unreachable, proceed to Section 4. Otherwise, continue.”
Assuming tool availability. If your monitoring dashboard is part of the outage, a step that says “Check Grafana” is dead weight. Include fallback commands that use local utilities: top, df -h, netstat.
Neglecting the human factor. Fatigue and stress make smart people sloppy. Build in a handoff trigger. A line as simple as “If incident duration > 90 min, escalate to secondary on-call” can prevent a bad night from turning worse.
Integrating with Existing Runbooks
Think of the recovery checklist as the front door. It should point to detailed runbooks for specific procedures. After the checklist directs a database failover, a link to the runbook for that exact database version and topology should sit right there. This keeps the checklist lean while the depth is a click away.
At the Gray Haven Lab, we cross-reference checklists with our infrastructure-as-code repos. When a Terraform module changes, the matching checklist gets reviewed in the same pull request. That coupling stops drift between the documented recovery path and what’s actually deployed.
Example Checklist Snippet: Web Application Recovery
Here’s a condensed example for a hypothetical web app. It shows the format and the level of detail you’re aiming for.
# Web App Recovery Checklist
# Trigger: Health check endpoint /health returns non-200 for 2 minutes
## Safety
[ ] Confirm no active deployment pipeline is running
[ ] Verify load balancer is not directing traffic to failed instances
## Triage
[ ] ssh to bastion host: ssh bastion.prod.example.com
[ ] Check application server status: systemctl status app-server
[ ] Tail recent logs: journalctl -u app-server --since "5 min ago"
## Stabilize
1. [ ] Restart app-server on primary node: systemctl restart app-server
Expected: /health returns 200 within 30 seconds
Rollback: revert to snapshot if restart fails
2. [ ] If primary restart fails, promote standby:
Run: /usr/local/bin/promote-standby.sh
Verify: curl -I https://app.example.com/health
## Communication
[ ] Update #incident-response with current status and ETA
## Post-Recovery
[ ] Run smoke test suite: /usr/local/bin/smoke-test.sh
[ ] Check error rates in monitoring for 10 minutes
[ ] Confirm backup cron is active: systemctl status backup-cron
Maintaining the Checklist Over Time
Systems shift. Dependencies drift. A checklist from six months ago might reference a server that’s been decommissioned or a flag that no longer exists. Schedule a review during each change management window. Assign an owner—a specific person, not a vague “team”—who’s responsible for verifying accuracy. At the Gray Haven Lab, checklist ownership rotates with the on-call schedule, which keeps fresh eyes on it and spreads the knowledge around.
FAQ
How is a recovery checklist different from a disaster recovery plan?
A disaster recovery plan is the big-picture document—business continuity, site failover, long-term restoration. A recovery checklist is tactical and immediate. It’s what an operator grabs first during an active incident, aimed at stabilization and triage rather than full recovery.
What if the checklist itself becomes unavailable during an outage?
This is a real risk, and it’s burned plenty of teams. Keep copies in multiple places: a Git repository with local clones on engineer laptops, a printed version in the data center, a PDF on a tablet stored offline. At minimum, stash a text file on a USB drive that doesn’t need a network to read.
How do I prevent the checklist from growing too long?
Set a hard limit of two pages for the core recovery steps. If a procedure needs more detail, link to an external runbook. Review the checklist quarterly and cut any step that hasn’t been used in the last three incidents. It’s a tool, not a museum piece.
Should non-technical staff be involved in checklist creation?
Yes, especially for the communication cadence section. Customer support leads and product managers can map out who needs to know what and when. Their input keeps the right stakeholder updates baked in, which cuts down on ad-hoc interruptions while you’re trying to fix things.
A recovery checklist is a quiet act of respect—for your future self, for your team, and for the people who rely on your systems. Write it now, while the server room hums and the dashboards stay green. That calm will seep into every line, and it’ll still be there when you need it most.

















