Incidents don’t knock. A drive that was fine yesterday suddenly isn’t. A typo in a config, a silent bit of data rot—and a Tuesday morning turns into a quiet scramble. At Gray Haven Lab, we’ve learned one thing over and over: the gap between a short outage and a long night often comes down to whether someone scribbled down the steps before the pressure started.
A recovery checklist isn’t an encyclopedia. It’s not meant to cover every weird edge case. It’s a short, ordered list of things a tired, stressed person can still follow at 3 a.m. This article is about building that list now—while the systems are humming—so when something breaks, you’re not making it up as you go.
Why Checklists Work When Your Brain Doesn’t
Under stress, working memory shrinks. Even seasoned operators skip steps, chase symptoms, or forget to check the obvious thing. A checklist takes that weight off your mind. It turns a jolt of panic into a series of small, checkable moves.
Atul Gawande’s The Checklist Manifesto showed how this plays out in surgery and aviation. Infrastructure isn’t that different. A checklist cuts down on errors of omission. For something like restoring a degraded ZFS pool or bringing a WireGuard tunnel back after a power loss, the checklist doesn’t teach you the commands. It just reminds you what to do, and in what order.
Checklist vs. Runbook vs. Documentation
These three get mixed up all the time. A runbook is the full guide—screenshots, explanations, branching logic. Documentation describes the system: IPs, architecture, what depends on what. A recovery checklist is the stripped-down version. It’s the critical path back to a known-good state, no commentary. It assumes you know the tech but might be too foggy to remember to unmount that dataset before importing it.
Decide What Gets a Checklist
Not every little service needs one. Start with the things whose failure hurts the most: the link to the outside world, primary storage, the identity provider that half the stack relies on. At the Lab, we map dependencies first—physical boxes, hypervisors, key VMs, gateways—and then ask: “If this one thing goes dark, what’s the shortest set of steps to bring it back?”
Typical candidates:
- Restoring a database from WAL backups after corruption.
- Rebuilding a boot mirror after a disk fails.
- Re-provisioning a router from a backup config when the firmware bricks.
- Recovering a DNS server that stopped resolving without a sound.
Shape the Checklist
A decent recovery checklist has a rhythm. It starts with a pre-check, moves through a straight sequence of actions, and ends with a check that proves the thing is actually back. Every line is an imperative: “Verify,” “Stop,” “Restore,” “Check.” No passive voice. The only branches are simple yes/no gates.
1. Pre-Flight Check
Before you change anything, confirm what’s broken. This bit should have commands or observations that isolate the failure. If a VM won’t start, the checklist might say: “Check hypervisor logs for qemu errors,” or “Ping the management interface.” The point is to avoid fixing the wrong thing while the right one stays broken.
2. The Recovery Sequence
This is the core. Each step is a single action you can verify. For a PostgreSQL recovery, it might go:
- Stop the PostgreSQL service.
- Move the corrupted data directory somewhere safe.
- Restore the base backup from the backup server.
- Apply WAL files in order.
- Start the service.
Include exact commands only when they cut through ambiguity. Don’t clutter the list with flags you can look up in a man page. The checklist is a pointer, not a manual.
3. Verification
The last section has to answer: “How do I know it’s working?” This might be a curl against an endpoint, a query that returns the rows you expect, or a ping to a known host. Without it, you might think you’ve fixed the problem when you’ve only hidden a symptom.
Write for the Tired Operator
Picture someone who just got woken up, is running on adrenaline, and might be typing one-handed while holding a phone. Short sentences. Plenty of white space. If a step has a known failure mode—“If the restore fails with error X, skip to step 7”—include it, but keep it tight. Bold or colored warnings can help, but don’t count on formatting that won’t survive a printout or a terminal.
At Gray Haven Lab, we test a checklist by handing it to someone who hasn’t touched the system in six months. If they can get through it without asking a single question, it passes. If they get stuck, we fix the list.
Keep It Alive
Checklists rot. IPs change. Backup paths move. Service names get renamed. Review them every quarter, or tie the review to any infrastructure change. A checklist that points to a backup server decommissioned last month is worse than nothing—it burns time and trust.
Store the checklist where it’s reachable even when the main systems are down. A text file on a jump host, a printed copy in a binder, a static page on an out-of-band management network. Just don’t keep the only copy on the system it’s meant to recover.
Dealing with Partial Failures
Not every incident is a full crash. Sometimes a service degrades: it answers slowly, drops a few requests, logs errors but stays up. Your checklist should acknowledge that. Add a short “partial failure” note at the top: “If the service is running but unhealthy, start at step 4 to reload the config without a full restart.” That keeps a well-meaning operator from pulling a full recovery when a lighter touch would do.
Build a Checklist Template
Make it repeatable. At the Lab, we keep a simple structure anyone can fill in:
- System name and description
- Pre-check commands
- Recovery steps (ordered)
- Verification steps
- Known failure modes
- Last reviewed date
The template lives in a shared repo and gets copied for each new critical service. Consistency helps: when you’re flipping through a binder at an odd hour, every checklist feels familiar.
FAQ
How is this different from a disaster recovery plan?
A disaster recovery plan is the high-level document about business continuity, communication, and multi-system restoration. A recovery checklist is granular and technical. It’s one small piece of that bigger plan, focused on bringing a single system back.
What if my setup changes all the time?
If your infrastructure is that dynamic, tie checklist updates to your config management. When a change gets pushed, make a checklist review part of the merge request. Even a two-minute sanity check keeps it from going stale.
Can I just use a wiki instead of a printed list?
A wiki is fine—if it’s available during an outage. If your wiki lives on the same hypervisor that just died, you’re out of luck. Keep an offline copy, printed or on a separate low-power device, for the stuff you really depend on.
How long should a recovery checklist be?
Shoot for one page. If it’s longer, you’re probably packing in too much context. Split gnarly recoveries into two: a primary path for common failures and an appendix for weird edge cases. The primary path is what you’ll grab first.
Practice the Checklist
An untested checklist is just a theory. Schedule a drill. Simulate a failure and work the steps. Time it. Note where you hesitated, where a command was missing, or where an assumption was wrong. At Gray Haven Lab, we run these drills in low-traffic windows and treat every stumble as a chance to revise.
Writing a recovery checklist is a small act of operational self-respect. It admits that humans, however sharp, do better with a bit of external structure when the clock is ticking. Take an hour this week, pick one critical system, and get it on paper. Your future, tired self will thank you.