It’s 2 a.m. and your database read replica is lagging so badly that customers are looking at yesterday’s data. Normally you’d pull the one engineer who knows the replication topology inside out into the war room. But tonight, they’re gone. Not “stepped away from their desk” gone—their Slack account is deactivated, their SSH keys are revoked, and the runbook you’re frantically scrolling through points to a script that lived in their home directory. For a team of five or six people, this isn’t a tabletop exercise. It’s the real thing, and it’s happening right now. Lean teams tend to accumulate single points of knowledge. You can’t fix that in the middle of a P1. What you can do is follow a handful of clear, unsentimental steps to keep the response moving when the person who built the thing is no longer in the room.
Immediate Triage: Lock the Doors First
Your hands will want to jump straight into the technical weeds. Don’t. The first danger isn’t the lagging replica—it’s that the departed engineer’s credentials are still floating around, and they could be used to make a bad situation catastrophic. Even if the split was perfectly cordial, their accounts are now a loose cannon. Revoke them.
Go straight to your identity provider and suspend the user. Then hit the cloud consoles: IAM roles, access keys, and any service-specific permissions. If you’re on AWS, for example, disable the access key and delete the login profile, but don’t remove the user or their policies yet—you might need to reconstruct what they had later. Check your VPN, your password manager, and any shared secrets they might have known. This is not about trust. It’s about shrinking the attack surface while the system is already bleeding.
If they had active SSH sessions on a bastion or a jump box, kill them. A simple pkill -u username can prevent an accidental—or deliberate—command from running in a moment of confusion. You’re not burning bridges; you’re closing a door that nobody should be walking through right now anyway.
Rebuilding the Map When the Cartographer Is Gone
With access locked down, you face the real gut punch: the mental model of the system left with the person. In a team of five, it’s common for one engineer to hold the deep lore about the database, the message queue, or the CI/CD pipeline. That lore is now inaccessible. You need to reconstruct enough of it to stop the bleeding.
Don’t try to understand everything. That’s a post-mortem activity. Right now, you’re looking for the narrowest thread that leads to the current failure. Start with the audit logs. Your cloud provider records every API call; your CI/CD system logs every deployment. Look for changes in the hours before the incident: a configuration push, a feature flag toggle, a manual command run against production. The trigger is almost always a recent change, and logs don’t leave with the engineer.
If the engineer used tmux or screen on a shared host, check for lingering sessions. I’ve found open log tails, half-finished commands, and once, a running htop that showed a memory leak in real time. Also, dig through Slack history and Notion drafts. Small teams often discuss changes informally before they become tickets. The clues are scattered, but they’re there.
Stabilize First, Understand Later
When you don’t have the full picture, the urge to start fixing things can backfire spectacularly. I’ve watched a well-meaning engineer restart a database they didn’t fully understand, turning a partial outage into a full data loss event. Adopt a read-only posture for as long as you can hold it. Your first job is to restore service, not to solve the root cause.
Can you fail over to a standby? Can you promote a read replica to take writes temporarily? Can you serve a static maintenance page while you triage? These are blunt instruments, but they buy you time. If you must make changes, make them tiny and reversible. Flip a feature flag to disable the broken component instead of rolling back a deployment whose process you’re unsure of. Take snapshots of databases, config files, and running processes before you touch anything. Those snapshots are your insurance policy—they give you a way back and they’ll be gold during the post-mortem.
In cloud-native environments, lean on the platform’s own tooling. On AWS, use Systems Manager Session Manager to get into instances without needing SSH keys. On Kubernetes, exhaust kubectl describe and kubectl logs before you ever type kubectl edit. The control plane is your new admin interface. Use it.
Breaking Glass: The Backup Admin Protocol
This is the moment you wish you’d set up a proper handoff procedure. If you have one, breathe. If you don’t, you’re about to invent one under fire. The protocol needs to answer three questions: Who has emergency access? What credentials do they need? How do they get them right now?
Ideally, you’ve got a sealed envelope somewhere—a physical safe, a shared 1Password vault with a recovery key, a hardware token in a locked drawer. Something that lets a secondary admin authenticate without the primary’s help. If that doesn’t exist, you’re going through your cloud provider’s account recovery process, and that can take hours or even days. Start it immediately, even if you think you won’t need it. You can always cancel it later.
For a more structured approach, refer to our Recovery Checklist guide, which outlines the exact steps to prepare a backup admin kit before you’re in the middle of an outage. Having that checklist ready transforms a chaotic handoff into a methodical verification.
Running the Incident with a Smaller Crew
Your incident command just lost a key player. Don’t let the gap fill itself—explicitly reassign roles. The person who was handling stakeholder updates might need to step into the technical lead spot. Someone else might need to be pulled from another team. Name names. This isn’t bureaucracy; it’s preventing the bystander effect. When everyone assumes someone else is handling the database, nobody handles the database.
Adjust your communication rhythm. Stakeholders will be nervous, and without the primary admin’s depth, your updates will be thinner. That’s fine. Tell them what you know, what you’re doing, and exactly when the next update is coming. Over-communicating uncertainty is far better than going dark while you investigate. Silence breeds speculation, and speculation makes people do unhelpful things.
If you have a vendor support contract for the affected system, use it early. The primary admin may have been the only person who ever opened a ticket, but most vendors have a process for emergency access by other authorized contacts. This is also the time to call any external consultants or partners who’ve touched your environment. Even partial familiarity can cut your diagnosis time in half.
After the Dust Settles: Closing the Gap for Good
Once the incident is resolved, the real work starts. A key engineer leaving during an outage is a blaring signal that your operational resilience has a single point of failure. The post-incident review shouldn’t just produce a timeline of the technical failure. It should produce a concrete plan to distribute the knowledge that walked out the door.
Start by writing down exactly what you did to recover. That becomes the skeleton of a new runbook. Then, list every system, credential, and configuration that only the departed engineer understood. For each one, assign a new owner and a deadline for them to pair with someone else and transfer that knowledge. The goal isn’t a perfect wiki—nobody reads those. The goal is to make sure at least two people have actually touched every critical component with their own hands.
Finally, update your on-call rotation and escalation policies. If your team is too small to have true redundancy for every system, say so explicitly. Document which systems have a bus factor of one, and set a recurring review to chip away at that list. This isn’t a one-and-done fix. It’s a commitment to making the team a little less fragile every quarter.
Frequently Asked Questions
What is the first thing I should do if the only engineer who knows a system leaves during an outage?
Immediately revoke or suspend their access credentials. This prevents any accidental or intentional changes that could worsen the incident. Focus on cloud provider IAM roles, VPN access, and shared secrets. Containment comes before diagnosis.
How can a small team prevent a single admin from being a critical failure point?
Practice paired operations for all critical systems. Every infrastructure component should have at least two people who have performed a deployment, rollback, and restore on it. Formalize this with a written backup admin protocol that includes emergency access procedures and is tested quarterly. See our guide on writing a recovery checklist before you need it for a step-by-step approach.
What if we have no documentation and the system is failing right now?
Focus on the narrowest path to stabilization. Use cloud provider audit logs and CI/CD history to identify the most recent changes. Adopt a read-only mindset: observe, do not modify, until you understand the change that triggered the failure. If you must intervene, take snapshots of everything first. Engage vendor support if available, as they can often provide architectural context you lack.
How do we handle stakeholder communication when we have lost our subject matter expert?
Be transparent about the situation without assigning blame. State that the primary engineer is unavailable, that you are following emergency procedures, and that you have engaged additional resources. Provide a clear, time-bound update schedule—even if the update is simply that investigation continues. Silence erodes trust faster than uncertainty.