Most disaster recovery plans carry a quiet, unspoken assumption: when the alert fires, someone is there to hear it. Someone awake, sober, near a keyboard, and not already buried under the same outage. That assumption sits so deep inside runbooks and escalation policies that it rarely gets questioned—until a 3 a.m. failure proves it wrong.

At Gray Haven Lab, we spend a lot of time thinking about the seams between infrastructure and human attention. Not the glossy, boardroom-ready version of resilience, but the operational truth of what happens when a critical system falls over and the designated responder is asleep, on a plane, or simply unreachable. The gap between the plan on paper and the plan at 0300 is where real downtime lives.

The Implicit Shift Schedule

Every recovery procedure has a timeline. Step one: detect the incident. Step two: notify the on-call engineer. Step three: begin diagnosis. The timeline assumes step two completes in seconds or minutes. But what if it takes forty-five minutes because the primary contact’s phone is on silent? What if the secondary contact is also the person who built the monitoring system—and that system is part of the failure?

This is not a staffing problem. It is a design problem. The plan treats human availability as a constant, like electricity or bandwidth. But humans are not always-on. They sleep, they get sick, they have lives outside the pager. A plan that does not account for the latency of human response is not a plan; it is a wish.

Consider the typical small-to-medium infrastructure team. There might be three people in the on-call rotation. The rotation looks solid on a calendar. But dig into the details: one person is a deep sleeper, another lives in a rural area with spotty cell coverage, and the third is often the one who notices problems first because they never fully disconnect. When the deep sleeper misses a page, the escalation lands on the rural engineer, whose phone rings to voicemail. By the time the always-connected person gets the alert, they have already been awake for an hour trying to fix it informally. The plan worked on paper. In practice, it failed twice before succeeding once.

Runbooks Written for Daylight

Runbooks are another place where the awake assumption hides. They are often written during business hours, by people who are fully caffeinated and have access to all the tools they need. The steps make sense when you can see the screen clearly, when your VPN token is handy, when you remember the exact command to check the message queue depth. At night, under stress, those same steps become obstacles.

A runbook that says “log into the admin console and check the dashboard” assumes the admin console is reachable. If the outage affects the identity provider, the console might refuse your credentials. If the outage is a network partition, the console might not load at all. The runbook rarely includes the step “first, verify you can authenticate.” It assumes you are already in.

We have seen runbooks that begin with “open a browser.” That is a daytime instruction. At night, on a phone, while half-asleep, opening a browser and navigating to a specific URL is a cognitive load that can take minutes. Multiply that by every step, and the recovery time stretches far beyond the RTO promised in the SLA.

The Single Point of Human Failure

Infrastructure teams obsess over single points of failure in systems. Redundant power supplies, clustered databases, multi-zone deployments. But the human in the loop is rarely treated with the same rigor. If only one person knows how to restart the legacy payment processor, that person is a single point of failure. If the entire recovery plan depends on a group chat that requires internet access, and the internet is down, the chat is a single point of failure.

This extends to knowledge. Tacit knowledge—the kind that lives in someone’s head and not in a wiki—is the most common single point of failure in recovery. The engineer who knows the quirks of the old storage array, the one who remembers the undocumented firewall rule, the one who can interpret the cryptic error message from the mainframe emulator. When that person is unavailable, the plan stalls. No amount of documentation can fully replace the mental model built from years of midnight fixes.

Escalation Paths That Dead-End

Escalation policies are supposed to be the safety net. If the primary doesn’t answer, the alert goes to the secondary. If the secondary doesn’t answer, it goes to the manager. But what happens when the manager is also the secondary? In small teams, role overlap is common. The escalation tree looks like a pyramid but is actually a single trunk with a few branches that all lead back to the same two people.

Even in larger organizations, escalation paths can dead-end in surprising ways. An alert escalates to a director who hasn’t been on-call in three years. They don’t have the current VPN client installed. Their credentials have expired. They call the original on-call engineer directly, bypassing the system, and now two people are working the problem without coordination. The plan assumed a clean handoff. Reality delivered chaos.

We recommend a practice we call “escalation path testing.” Once a quarter, simulate an alert at an inconvenient time—not a full outage, just a page—and see how long it takes for a human to acknowledge it. Do not announce the test in advance. The results are often sobering. One team we worked with discovered their primary on-call phone number was routing to a decommissioned handset. The alert had been going nowhere for six months.

When the Monitoring Sleeps Too

Monitoring systems themselves are not immune to the awake assumption. Many monitoring setups rely on a central server that runs checks and sends alerts. If that server goes down, it stops checking. It also stops alerting. The failure is silent. The team wakes up in the morning to discover the outage by its effects, not by its alarm.

Dead man’s switches can help, but they are rarely implemented. A dead man’s switch is a separate, minimal system that expects a regular heartbeat from the main monitoring system. If the heartbeat stops, the switch triggers an alert through a completely independent channel—a different cloud provider, a different messaging service, even a hardware pager if you can still find one. The switch does not need to diagnose the problem. It only needs to say “the thing that watches the things has stopped watching.”

This is an example of designing for unavailability. Instead of assuming the monitoring system will always be up, you assume it will eventually fail and build a detector for that failure. The same principle applies to every component in the recovery chain. Assume the on-call phone will be off. Assume the chat system will be unreachable. Assume the person with the root password is on vacation. Then build around those assumptions.

Designing for the Unavailable Responder

So what does a recovery plan look like when it stops assuming someone is awake? It starts with a shift in mindset: from “who will fix this” to “how will this get fixed if no one is available.” That shift leads to concrete design changes.

Pre-approved automated responses. For certain well-understood failure modes, the system can take action without human approval. If a web server stops responding, an auto-healer can restart it. If a disk fills up, a script can rotate logs. The key is to define the blast radius carefully. Automation that can cause data loss or cascading failures needs a human gate. But automation that restores a known-good state is safer than waiting forty minutes for a human to do the same thing.

Out-of-band communication channels. If the primary chat system is down, the team needs a backup that does not depend on the same infrastructure. This could be a separate Slack workspace on a different cloud provider, a group SMS list, or even a phone tree. The backup channel must be tested regularly, and its contact information must be stored somewhere that does not require the primary infrastructure to access. A printed card in a wallet is more reliable than a wiki page during a network outage.

Runbooks written for exhaustion. Recovery procedures should be written as if the reader is tired, stressed, and working from a phone. Use large, clear headings. Put the most critical diagnostic commands at the top. Include the exact commands to run, not descriptions of what to check. “Run systemctl status nginx” is better than “verify the web server is running.” Include fallback steps for when the primary tool is unavailable. If the admin console is down, what API call can you make instead? If the VPN is broken, is there a jump host you can reach directly?

Shared, searchable knowledge. Tacit knowledge must be externalized. This does not mean writing exhaustive documentation—that is a trap that leads to stale, unread wikis. It means creating a searchable log of past incidents, with the actual commands run and the actual observations made. When the expert is asleep, the responder can search for symptoms and find what worked last time. This is a living document, updated during or immediately after every incident. We wrote about this approach in Write the Recovery Checklist Before You Need It, which covers how to build checklists that work under pressure.

Rethinking On-Call Rotations

The traditional on-call rotation assumes that one person is always reachable. A more resilient model assumes that no single person is always reachable, and designs accordingly. This might mean overlapping shifts where two people are primary during high-risk hours. It might mean a follow-the-sun model where the pager moves across time zones so no one is woken up. It might mean a tiered response where level-1 triage is handled by a larger pool of people who only need to diagnose and escalate, not fix.

None of these models eliminate the awake assumption entirely, but they reduce the probability that all responders are simultaneously unavailable. They also reduce the cognitive load on any single person, which improves decision quality when someone does respond. A well-rested engineer who is not on their sixth consecutive night of call is more likely to follow the runbook correctly and less likely to make a costly mistake.

It is also worth examining the alerting thresholds themselves. Many alerts are set too sensitively, generating pages for conditions that do not require immediate human attention. Each false page erodes the responder’s trust in the system and increases the chance they will sleep through a real one. Tuning alerts so that only actionable, time-critical conditions wake someone up is a form of respect for human attention. It acknowledges that sleep is a finite resource and should not be spent on non-urgent problems.

The Cost of the Awake Assumption

When a recovery plan assumes someone is awake, the cost of that assumption shows up in two places: the duration of the outage and the health of the team. The duration cost is obvious. Every minute spent waiting for a human to wake up, find a laptop, and orient themselves is a minute the system is down. For customer-facing services, that translates directly to revenue loss and reputational damage. For internal systems, it means delayed decisions, blocked workflows, and a cascade of secondary failures.

The human cost is less visible but more corrosive. On-call engineers who are repeatedly woken for false alarms or poorly designed escalations burn out. They leave. Their replacements inherit the same flawed plans and the same sleep deprivation. The cycle continues until the team is a skeleton crew of exhausted people who have learned to sleep through alerts because most of them don’t matter. That is not resilience. That is a system slowly training its humans to ignore it.

There is also a subtler cost: the erosion of trust between the people who design recovery plans and the people who execute them. When a plan assumes availability that doesn’t exist, the responders feel set up to fail. They know the RTO is unrealistic. They know the escalation path is broken. They stop believing in the plan, and once belief is gone, the plan becomes a formality—something to show auditors, not something to follow.

Building Plans for the Real World

A recovery plan that works at 3 a.m. looks different from one written at 10 a.m. It is shorter. It is more direct. It assumes the responder is impaired by fatigue and stress. It includes checkpoints where the responder can verify they are on the right track. It has clear criteria for when to stop troubleshooting and escalate to a wider group—even if that means waking up the entire team.

One practical technique is to write the plan in two columns. The left column is the ideal path: what to do when everything works. The right column is the degraded path: what to do when the usual tools are unavailable. For example, if the ideal path says “open the dashboard at https://admin.example.com,” the degraded path says “if the dashboard is unreachable, SSH to the host and run systemctl status.” This small addition acknowledges that the responder might be working from a phone, over a slow link, without access to the normal web interfaces.

Another technique is to include a “first five minutes” section at the top of every runbook. This section does not try to fix the problem. It focuses on orientation: what system is affected, who is the current on-call, what are the immediate safety checks to prevent the problem from spreading. The goal is to give the responder a foothold before they dive into diagnosis. In the middle of the night, that foothold can prevent a cascade of mistakes.

Testing the Unavailable Responder Scenario

Most disaster recovery tests assume full human availability. The team gathers in a conference room or a video call. Everyone is present, focused, and ready. The test validates the technical steps but ignores the human latency that will be present in a real incident. A more honest test would simulate the unavailability of key people. Run the test with the primary on-call “unreachable.” See how long it takes for the secondary to engage. See if the runbook still makes sense when the person executing it has never done it before.

These tests will reveal gaps that tabletop exercises miss. They will show that the secondary on-call doesn’t have access to the right AWS account. They will show that the runbook references a dashboard that was decommissioned last month. They will show that the “emergency contact list” is stored on a wiki that requires VPN access, and the VPN credentials are only on the primary’s laptop. Each of these gaps is a single point of failure that the plan assumed away.

Fixing these gaps is not expensive. It requires time and attention, not capital. It requires someone to walk through the plan from the perspective of a tired, under-equipped responder and ask “what would stop me?” It requires the team to accept that their plan is fragile and to treat that fragility as a bug to be fixed, not a reality to be accepted.

FAQ

Why do recovery plans assume someone is awake?

Recovery plans are typically written during normal business hours by people who are fully alert and have access to all necessary tools. The writers unconsciously project their own state onto the imagined responder. Additionally, many plans are designed to meet compliance requirements rather than operational reality, so the human factors are overlooked in favor of documenting technical steps.

How can we test if our plan works when no one is available?

Conduct unannounced tests during off-hours where the primary on-call is instructed not to respond. Measure how long it takes for the secondary to acknowledge the alert and begin diagnosis. Observe whether the runbook is usable under fatigue conditions. These tests should be done quarterly and the results used to update the plan and fix discovered gaps.

What is the most common single point of human failure in recovery?

Tacit knowledge—information that lives only in someone’s head—is the most common single point of failure. When the person who knows the undocumented quirks of a system is unavailable, the recovery stalls. Externalizing this knowledge through incident logs and searchable runbooks reduces the dependency on any single individual.

How do we write runbooks that work at 3 a.m.?

Write runbooks as if the reader is tired, stressed, and working from a phone. Use clear headings, provide exact commands rather than descriptions, and include fallback steps for when primary tools are unavailable. Add a “first five minutes” section focused on orientation and safety checks. Maintain a two-column format with ideal and degraded paths.

Disaster recovery is not just a technical discipline. It is a human one. The best infrastructure in the world will not recover itself if the person who knows how to start the recovery is unreachable. By designing plans that assume unavailability—of people, of tools, of attention—we build systems that are resilient in the hours when resilience matters most.