Recovery plans tend to get scribbled out in the fog after something breaks. That is exactly the wrong moment. Systems are down, everyone is on edge, and clear thinking is hard to come by. What you actually want is a checklist you can reach for and follow with half a brain—something already tested and waiting. At Gray Haven Lab, we treat recovery documentation as quiet, ongoing operational work, not a last-minute scramble. A solid checklist is a technical artifact, shaped by your real infrastructure, your real dependencies, and what your team can actually pull off when things go sideways. Here is how to build one before you ever need it.
Start with a Dependency Map, Not a Wish List
Do not write a single step until you know what your systems actually lean on. I am not talking about a pretty diagram of what you think is there. I mean a verified account of what runs where, what talks to what, and what falls over when something else stops. A dependency map shines a light on the single points of failure your checklist has to tackle first.
Start at the user-facing edge and trace inward. For a typical web app, follow the path from the DNS provider through the CDN, to the load balancer, application servers, database clusters, and any external APIs you rely on. Record the version, config, and failover status for every piece. Use real discovery, not assumptions: dig for DNS, poke around cloud provider consoles, and comb through your IaC definitions. When we do this at Gray Haven Lab, we almost always dig up some forgotten service nobody remembered—until it broke.
Once you have the map, slap a worst-case recovery time on each node. Not a formal SLA, just an honest estimate based on the last time someone rebuilt it from the ground up. If nobody has ever rebuilt it, the estimate is “infinite.” That tells you exactly where your checklist needs the most meat.
Write the Checklist in a State of Calm
Writing documentation in the middle of an incident gets you something reactive and full of holes. The right time is during a maintenance window, a routine drill, or any stretch where nothing is actively burning. The person writing it should be the one who would do the recovery—or someone who already has. A checklist dreamed up by an architect who never touches production will miss the gritty, real-world steps every time.
Lay out the checklist as a straight line of actions, with clear preconditions and postconditions. Instead of “Restore database,” get specific:
- Precondition: Backup file available at
/mnt/backups/db-prod-YYYYMMDD.sql.gz. - Action: Run
pg_restore --clean --dbname=app_prod /mnt/backups/db-prod-YYYYMMDD.sql.gzon the primary node. - Postcondition: Verify row count on
userstable matches pre-incident baseline (1.2M rows).
This kind of detail cuts through ambiguity when the pressure is on. It also exposes gaps. If you can’t nail down the precondition, you haven’t really tested the backup pipeline. If the postcondition reads like a shrug, you don’t know what “healthy” actually means.
Design the Checklist for Human Operators
A recovery checklist is not a script to be run. It is a guide for a person whose working memory is shot because everything is urgent. Use short, imperative sentences. Group related steps under clear headings. Avoid conditional logic that forces someone to think—swap “If the primary fails” for “Primary failed: go to Section B.”
Drop in verification points every few steps. A verification point is a quick, non-destructive check that the last action did what you expected. After restarting a service, verify it is listening on the right port and logging like normal. These little checkpoints stop errors from snowballing.
Think about the physical world, too. If recovery means on-site hardware, list where the keys, console cables, and crash carts live. If it needs a password manager, include the exact path to the credential entry. In an earlier post, we dug into the broader mindset of preparing recovery docs with a calm, methodical approach: Write the Recovery Checklist Before You Need It. The same logic fits here—every detail you skip becomes a decision someone has to make under duress.
Test the Checklist Against Real Failure Modes
A checklist nobody has tested is just a theory. Testing does not mean a cozy tabletop walkthrough. It means making the failure happen and following the steps. Begin with low-risk stuff, like a read-replica outage, and work up to total primary database loss. Each round will uncover missing steps, stale commands, and wrong assumptions about system state.
Write down what you find. If a step took three times longer than you guessed, update the checklist with a real time estimate. If a verification command spit out something unexpected, fix the check. After every test, sit down briefly with the operator who ran it. Their firsthand friction is the best feedback you will ever get.
Do these tests on a schedule. Infrastructure drifts: certificates expire, IPs shift, tooling changes. A checklist that was fine six months ago might fail today because someone yanked a deprecated CLI flag. Hook the tests to your maintenance calendar, not to post-incident retrospectives.
Keep the Checklist Accessible When Systems Are Down
Even the finest checklist is worthless if nobody can get to it. If your one copy lives on a wiki hosted on the same infrastructure that just died, well, you already know the problem. Store the checklist in at least two independent spots: a printed binder in the ops center, a static HTML page on external object storage, or a markdown file in a Git repo mirrored to a different provider.
Think through the access chain. Does the operator need to log in to reach the checklist? If the identity provider is also down, that login is a dead end. That is why one copy should sit somewhere that needs no authentication—a laminated two-page card or a plaintext file on a USB stick kept in a secure physical spot. Format counts, too: plain text or PDF renders on anything, while a fancy web app might choke on a stripped-down recovery terminal.
Integrate the Checklist into Your Incident Response Workflow
A recovery checklist does not float in a vacuum. It is one piece of a bigger incident lifecycle: detection, diagnosis, containment, recovery, postmortem. The checklist handles the recovery phase, but it has to connect cleanly to what comes before and after. At the top of the checklist, add a short preamble stating the trigger. Something like: “Start this checklist when the primary database cluster stays unreachable for more than 5 minutes, confirmed by monitoring alert DB_PRIMARY_DOWN.”
At the tail end, build in a handoff section. Spell out the steps to shift from recovery back to normal operations: re-enable monitoring alerts, notify stakeholders, schedule a post-incident review. This sidesteps the all-too-common slip where systems come back but alerting stays silent, leaving the team blind to whatever hits next.
Maintain the Checklist as a Living Document
Recovery checklists rot. Every infrastructure tweak, every person who leaves, every software update can break pieces of it. Pin ownership to a team or role, not one person who might move on. The owner’s job is to review and refresh the checklist after any meaningful production change, and to make sure the scheduled tests actually happen.
Version it. Stick a simple revision number and date at the top, and keep a changelog of what changed and why. This builds accountability and makes it easy to roll back if a new version introduces a mistake. Treat the checklist with the same respect you give your infrastructure-as-code repos: changes go through review, and the latest version is the one you would trust when things are on fire.
FAQ
Who should write the recovery checklist?
The person or team on the hook for doing the recovery. Usually that is a site reliability engineer, a senior ops specialist, or a developer with deep systems knowledge. The real requirement is hands-on experience with the production environment—not just architectural diagrams.
How often should the checklist be tested?
At least every quarter, and after any major infrastructure change. High-risk systems might need monthly tests. Match the cadence to the rate of change in your world: a fast-moving startup shipping daily needs more frequent testing than a stable, long-lived deployment.
What if a step in the checklist fails during recovery?
Each step should carry a fallback or escalation path. If a restore command bombs, the checklist might say: “If restore fails, check disk space on target node. If that is tight, escalate to storage engineering via channel #storage-escalation.” The whole point is to keep a single operator from having to improvise alone.
Can the checklist replace runbooks and playbooks?
No. A recovery checklist is narrowly focused on restoring service after a failure. Runbooks cover routine operations, and playbooks handle the broader incident response flow. The checklist sits alongside those docs, not in their place. They should link to each other wherever it makes sense.