Gray Haven Lab

A break-glass account is a pre-staged emergency credential. Its value is not that it logs in fast. Its value is that when it logs in, everyone knows, and the session can be traced back to a named human afterward. For a team of two to fifteen engineers who run their own on-call rotation, that distinction matters more than it does in an organization with a dedicated security operations center.

The failure mode the account exists for is specific: the identity provider is down, the MFA device is lost, the last global administrator has left, or the federation path is unreachable. Microsoft’s guidance on emergency access accounts in Entra ID names these cases directly and states the rule plainly: use emergency access accounts only for emergency or break-glass scenarios where normal administrative accounts cannot be used, and restrict their use to times when it is absolutely necessary. The same logic applies whether your control plane is Entra ID, a self-hosted LDAP directory, or a cloud provider’s IAM.

Design constraints for a small team

Microsoft recommends creating two or more emergency access accounts, as cloud-only accounts on the *.onmicrosoft.com domain that are not federated or synchronized from on-premises. The redundancy is not decorative. A single emergency account is a single point of failure that happens to be the thing you reach for when everything else has already failed.

The federation independence is the load-bearing part. If your emergency account authenticates through the same identity provider that just went down, you have not built an emergency path. You have built a second door with the same broken lock. For a bare-metal or self-hosted environment, the equivalent is a local account that does not depend on the central directory, with credentials stored somewhere that does not require the network to reach.

Microsoft also specifies that emergency access accounts should be excluded from Conditional Access policies that block or restrict sign-in, because an enforced policy could prevent sign-in during the exact emergency the account is designed for. Report-only policies do not block access and do not need an exclusion. This is a tradeoff worth stating plainly: you are deliberately carving a hole in your normal access policy. The hole is the control. It is not a bug to be closed later.

The authentication method for the emergency account should differ from the methods used by normal administrative accounts. If your normal admin accounts use an authenticator app, the emergency account should use a FIDO2 security key or certificate-based authentication. The credential or device must not expire or fall under automated cleanup for lack of use. An emergency account that gets swept by a dormant-account policy is an emergency account that will not work when you need it.

Make the account conspicuous by design

Yale’s HIPAA break-glass procedure, which draws on a 2004 whitepaper from the Joint NEMA/COCIR/JIRA Security and Privacy Committee, describes the design principle well: emergency accounts should be created in advance so that careful thought can go into the access controls and audit trails associated with them, and the username should be obvious and meaningful, such as breakglass01, so that the account name would be inappropriate under normal operations and would stand out in audit trails.

That is the opposite of how most credentials are named. Normal accounts blend in. The emergency account should not. When someone greps the audit log six months from now, the account name should be the first thing that catches their eye.

The distribution method should also be physically evident. Yale’s procedure lists several options: kept behind glass in a cabinet where access requires literally breaking the glass, maintained within sealed envelopes where a broken seal is an obvious indication of access, locked in a desk drawer that only specific people can access, or locked in a safe or cabinet where one person knows the combination and a different person has the key to the room. The common thread is that access to the credential leaves a physical trace.

For a small team, the two-person pattern is practical. One engineer holds the safe combination; another holds the office key. Neither can reach the credential alone. The friction is the point. It is not a bug to be optimized away.

The distribution procedure should include a sign-out method requiring an acceptable form of identification, recorded before the accounts are made available. Yale’s procedure notes that this assures activities performed using the emergency account may eventually be associated with an authorized individual, creates accountability, and can assure non-repudiation. In practice, that means a paper log or a shared document with a timestamp, a name, and a reason. The reason field is not bureaucracy. It is the first line of the postmortem.

Wire the alert before you need it

Microsoft’s guidance is direct: monitor sign-in and audit log activity from the emergency accounts and trigger notifications to other administrators, so that you can verify these accounts are only used for testing or actual emergencies. The alert is the account’s real security control. If a break-glass sign-in does not page a human within minutes, the audit trail is archaeology rather than detection, and the account is effectively a permanent backdoor with a nice name.

For a team without a SOC, the alert should route to the same on-call channel that handles production incidents. The alert should fire on every sign-in, not on a threshold. There is no legitimate reason for a break-glass account to sign in without a human knowing about it in real time.

The audit record itself should contain enough to reconstruct the event. NIST SP 800-53 Rev. 5, as reflected in the CSF reference, specifies that audit records should establish what type of event occurred, when it occurred, where it occurred, the source of the event, the outcome, and the identity of any individuals, subjects, or objects associated with the event. For a break-glass session, that means the account name, the source IP or workstation, the timestamp in UTC, the actions taken, and the outcome. If your system cannot produce that, Yale’s procedure is explicit: break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead.

NIST SP 800-92 frames log management as the process for generating, transmitting, storing, analyzing, and disposing of computer security log data, and notes that it is essential to ensuring that computer security records are stored in sufficient detail for an appropriate period of time. For a small team, the practical version is: ship the break-glass account’s logs to a destination that the account itself cannot modify, and keep them long enough to cover your incident review cycle.

Write the post-use runbook

After any use of an emergency access account, Microsoft’s guidance calls for a review to determine whether the use was authorized and whether the actions taken were appropriate. That review has a few concrete steps.

First, preserve the logs. Capture the sign-in record, the audit trail of privileged actions, and any session recordings or command history the platform provides. NIST’s audit guidance specifies that audit record reduction and report generation should support on-demand review and after-the-fact investigations without altering the original content or time ordering of audit records. Copy the raw records before you summarize them.

Second, reconcile the session to a named human. The sign-out log from the distribution procedure gives you the name. The audit trail gives you the actions. The postmortem gives you the reason. If those three do not line up, that is the finding.

Third, disable or rotate the credential. Yale’s cleanup procedure recommends disabling or deleting the emergency account that was used to prevent reuse now that the password is known, and reconciling the data acquired and audit trails to reflect the proper operator’s name. Some systems can automatically deactivate emergency accounts after first use or after a selectable period such as 8 hours or 1 day. Commvault’s break-glass implementation takes a different approach: the password is regenerated on every login to prevent repeated use of the same password, and all actions are recorded in the audit trail. Either pattern works. What does not work is leaving a known password in place after an incident that involved a dozen people in a channel.

Fourth, ask whether the normal access path failed in a way that should be fixed. Yale’s procedure notes that frequent use of emergency accounts may indicate problems with the normal user authentication mechanism. A break-glass account that gets used every month is not a break-glass account. It is a helpdesk with extra steps, and Yale’s procedure explicitly says break-glass should not replace a helpdesk.

Schedule the drill

Microsoft recommends performing account validation drills at least every 90 days, including reviewing the authorized-user list and testing sign-in and administrative task functionality. The 90-day cadence is a reasonable default for a small team. The important part is the second half of that sentence: testing administrative task functionality, not just sign-in.

A drill that only tests sign-in is not a drill. The failure that matters is discovering at 3 a.m. that the credential works but the permission it grants does not cover the task. If the break-glass account is supposed to let you restore a database, the drill should include a restore. If it is supposed to let you rotate an IAM role, the drill should include a role switch. AWS IAM documentation notes that when you switch to a role in the console, you temporarily give up your user permissions and work with the permissions assigned to the role, and when you exit the role, your user permissions are automatically restored. That is a useful property for a break-glass path, but only if the role’s permissions actually cover the emergency task.

One more AWS-specific detail worth knowing: you cannot switch roles in the AWS Management Console to a role that requires an ExternalId value. Such a role can only be assumed by calling the AssumeRole API that supports the ExternalId parameter. If your break-glass path depends on a role with an ExternalId, the console switch will not work, and the drill will find that out before the incident does.

Treat a failed drill as an incident. It is a control that did not work when tested under controlled conditions. The blameless postmortem format applies: what did we expect, what happened, what will we change.

The tradeoff to accept

The break-glass account is slower and more awkward than a normal login on purpose. Two people, a physical envelope, a sign-out log, a page to the on-call channel, a postmortem the next morning. Every one of those steps is friction, and every one of those steps is the control.

The alternative is an account that is fast and quiet. That account is not a break-glass account. It is a standing privilege with a reassuring name, and the first time it is used without anyone noticing, you will have lost the ability to tell an emergency from a misuse.

If you are building the recovery checklist that the break-glass account supports, the Recovery Checklist Before You Need It is a useful companion. The break-glass account gets you into the system. The checklist tells you what to do once you are there.

Questions that come up

How many break-glass accounts should a small team have? Microsoft recommends two or more for redundancy. For a team of two to fifteen, two is the practical minimum. One is a single point of failure. More than two multiplies the credential storage and rotation burden without adding much resilience.

Should the break-glass account be a person or a shared account? Microsoft’s guidance says not to associate emergency access accounts with any individual user in the organization, and to store credentials in a known secure location available to multiple members of the administration team. The account is a shared emergency path, not a personal login. The sign-out log is what ties a specific session to a specific person.

What if the system cannot produce an audit trail? Yale’s procedure is explicit: if the system cannot provide an audit trail showing simple account activity like login attempts, break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead. A break-glass account without an audit trail is a standing privilege with no accountability.

How often should the credential be rotated? The sources describe per-login regeneration (Commvault) and post-use disable-or-delete (Yale), but do not specify a fixed rotation interval for offline-stored credentials. The 90-day validation drill is the cadence the sources support. Rotation after every use is the safer default; a fixed calendar rotation is a secondary control.

Does the break-glass account replace normal access controls? No. The sources frame it as a last resort alongside standard controls. Yale’s procedure says standard access controls should be established with sufficient rules to minimize the number of times break-glass needs to occur. The account is the exception path, not the primary one.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The garment preview shows one word: meh. Lowercase. Bold. White on a dark shirt. No animal. No subtext. No parenthetical. The period is the only punctuation, and it lands like a closed door.

I keep looking at it because it refuses to do the thing most slogan shirts do. It doesn’t ask a question. It doesn’t list a hobby. It doesn’t wink at a fandom. It states a verdict and stops. For a team that spends its days writing runbooks, escalation policies, and decision records, that restraint is worth studying.

The shirt is listed as Meh Funny Sarcastic Saying Introvert Dry Humor under the Quirky Cotton Collective brand. The retail title stacks three descriptors — funny, sarcastic, introvert, dry humor — but the artwork itself carries none of that scaffolding. The design is the word. Everything else is metadata.

What the artwork actually does

Large bold lowercase typography on a dark ground is a visual choice with consequences. Lowercase reads as casual, unbothered, almost mumbled. Bold weight makes it loud. White on dark gives it contrast. The combination is a contradiction: a shout that says nothing is worth shouting about.

That contradiction is the joke, if there is one. The word meh is a shrug. Rendering it in the largest available type is a way of shrugging theatrically. The shirt doesn’t claim the wearer is indifferent to everything; it claims the wearer has decided this particular moment — the moment of reading the shirt — does not merit enthusiasm.

I’m not going to pretend this is a deep statement about labor, burnout, or the human condition. It’s a word on a shirt. But the word is doing something specific: it is refusing to promise anything. And that refusal is the most interesting thing about it.

Slogans versus promises

In operations work, we spend a lot of time distinguishing between a slogan and a usable promise. A slogan is a phrase that sounds good in a meeting. A usable promise is a sentence you can test. “We prioritize reliability” is a slogan. “We page a human within five minutes of a failed synthetic check” is a usable promise. One of them survives an incident; the other becomes a postmortem bullet point.

The meh shirt is not a usable promise. It is not trying to be. It is a slogan that knows it’s a slogan. It doesn’t claim the wearer is calm under pressure, or that they have a five-nines uptime record, or that they never lose their temper in a code review. It claims they are unimpressed. That’s a mood, not a metric.

I find that honesty refreshing. Most slogan shirts overpromise. They imply a personality trait — “I’m the funny one,” “I’m the smart one,” “I’m the one who fixes your computer” — and then the wearer has to live up to it. The meh shirt implies nothing. It is a preemptive withdrawal from the conversation. You can’t be disappointed by someone who never claimed to be impressive.

The introvert framing

The retail title includes “Introvert,” and the related search queries include “meh introvert shirt.” I understand why. The word meh functions as a boundary marker. It says: I am here, I am participating at the minimum required level, and I would like to go home now. For people who find social interaction draining, that is a recognizable posture.

But the artwork itself does not say “introvert.” It says meh. The introvert reading is an interpretation, not a printed claim. I think that’s important. The shirt is not a diagnosis. It is a mood that happens to align with a certain temperament. You can wear it as an introvert, an extrovert, or someone who just doesn’t want to talk about the weather. The word doesn’t check your credentials.

In team settings, this matters. A shirt that says “I’m an introvert” invites a conversation about personality types. A shirt that says meh invites nothing. It closes the loop. For someone who wants to signal a boundary without explaining it, that’s efficient.

What it doesn’t do

I want to be clear about what this shirt is not. It is not a diagnostic tool. It does not measure burnout, introversion, or operational fatigue. It does not cure anything. It does not improve team communication. It is a piece of clothing with a word on it.

I also can’t tell you how it fits, how it wears, or how it holds up after a wash. I haven’t worn it. I’m looking at the artwork as observed in the native garment preview: large bold white lowercase meh. lettering on a dark shirt, no animal illustration present. That’s the object. Everything else is inference.

What I can say is that the design is legible. The word is the whole design. There is no competing element, no decorative flourish, no hidden meaning. It is a single word, rendered large, and it does not pretend to be more than that.

Why this matters for technical readers

If you work on a small team, you are probably familiar with the gap between what a system promises and what it delivers. You write a runbook that says “restore from backup,” and then you discover the backup is corrupt. You write an escalation policy that says “page the on-call,” and then you discover the on-call is asleep. The gap between slogan and promise is where incidents live.

The meh shirt is a reminder that not everything needs to be a promise. Some things are just statements. Some things are just moods. Some things are just a word on a shirt. And that’s fine. Not every object has to carry the weight of a reliability practice.

But I do think there’s a small lesson here about honesty in language. The shirt doesn’t overclaim. It doesn’t say “I’m fine.” It doesn’t say “I’m thriving.” It says meh. That’s a low bar, and it clears it. In a world of overpromising slogans, there’s something to be said for a garment that promises nothing and delivers exactly that.

What we’d do differently

If we were designing a shirt to communicate a boundary, we would probably not choose a single word. We would choose a sentence with a subject and a verb, something testable. But that would be a different object. The meh shirt is not a boundary statement; it is a mood statement. And moods don’t need to be testable.

What we would do differently is resist the urge to over-interpret it. It is a word on a shirt. It is not a manifesto. It is not a diagnosis. It is not a promise. It is a shrug, rendered in bold lowercase type, and it is content to be exactly that.

Sometimes the most honest thing you can print is the thing you don’t bother to capitalize.

A runbook can be correct when written, still linked from the alert that invokes it, and wrong in the one step that matters. For a team of two to fifteen engineers who are their own on-call rotation, that is the expensive version of staleness: the document looks authoritative, the alert keeps firing, and the person reading it at 3 a.m. has no way to know which paragraph has quietly stopped matching production.

This is not a documentation problem in the abstract. It is a detection problem. Staleness is a property of the relationship between a runbook and the system it describes, not of the runbook’s age. A six-month-old runbook for a service that has not changed may be perfectly fresh. A two-week-old runbook for a service that shipped a new deploy path may already be wrong. The goal is to make that decay surface as a routine chore on a Tuesday afternoon rather than as a surprise during a customer-visible outage.

Why staleness is invisible by default

Runbooks are read under pressure, not on a schedule. The alert that points to one keeps firing, which creates a false sense of validation: the runbook is still reachable, so it must still be right. Nothing in the normal workflow compares the document to the system it describes. The person who wrote it may be the only person who has ever executed it, and their memory silently fills the gaps a fresh reader would hit immediately.

Google’s SRE Workbook is direct about the underlying rate of change. In its chapter on on-call, it states that playbook details go out of date at the same rate as the production environment changes, and that for daily releases a playbook might need an update on any given day. The same chapter notes that in SRE, whenever an alert is created, a corresponding playbook entry is usually created, and that these guides reduce stress, mean time to repair, and the risk of human error. That is the case for having runbooks. It is also the case for not trusting them indefinitely.

The workbook offers two broad strategies for keeping playbooks maintainable. One is to keep entries general enough that they change slowly — for example, a single entry for a class of alerts that a trained on-call engineer reads alongside an architecture diagram. The other is to write step-by-step procedures that reduce human variability and drive down MTTR. Both are legitimate. The workbook’s practical advice is that if a team agrees on nothing else, it should decide what minimal, structured details its playbooks must have, and notice when playbooks have accumulated a lot of information beyond those structured details. That minimal structured set is also the natural place to hang freshness signals.

Borrow the postmortem discipline

Google’s postmortem chapter contains a line that applies almost verbatim to runbooks: an unreviewed postmortem might as well never have existed. The chapter encourages regular review sessions so that each completed draft is actually reviewed, and it stresses that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary. It also lists a monitoring failure — which usually implies manual incident discovery — among the common triggers.

The same logic transfers. Define the runbook review trigger before you need the runbook. If the only time a runbook gets read is during an incident, then the incident is your review process, and you are paying for it in customer impact. A scheduled freshness check is cheaper.

Staleness signals worth checking

The following checklist is our own operational reasoning, not a measured result. It is offered as a template. The signals are chosen because each one can be checked mechanically or by a reader who did not write the runbook.

• Last-verified date and named owner. A runbook without an owner has no one to notice when it drifts. A date without a name is a timestamp, not accountability. A vendor guide on incident response runbooks recommends scheduling quarterly reviews, making version control standard, and keeping ownership visible so the runbook reflects the current state of the infrastructure. The cadence is a recommendation, not a finding; the ownership point is the durable part.
• The alert or dashboard the runbook is bound to still exists and still fires. If the alert was renamed, retired, or split, the runbook is orphaned even though its text is unchanged. Orphan detection is a mechanical check and can be automated with a link and asset linter.
• Every command still parses and its expected output still matches. A command that returns a different shape of output than the runbook says to expect is a stale runbook, even if the command itself succeeds.
• Every linked dashboard, log query, and escalation path still resolves. Dead links are the cheapest staleness signal to detect and the easiest to ignore.
• The rollback step still corresponds to the current deploy mechanism. Deploy paths change more often than most other parts of a service, and the rollback step is the one you least want to discover is wrong.
• The stated blast radius still matches the service’s current dependencies. A runbook that says “affects the checkout service” when the service now also fronts authentication is understating the incident.

One more signal is worth treating as a smell in its own right: if the runbook is a fixed list of commands run every time a particular alert fires, that is a sign the runbook is standing in for automation. The SRE Workbook makes this explicit — if playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, it recommends implementing automation. A runbook that should be a script will drift every time the underlying command surface changes, and it will drift silently.

A cheap freshness drill for a small team

Pick one runbook per week. Have someone who did not write it execute it against a non-production or read-only path. Record what broke. That is the whole drill.

This is a rehearsal of the document, not of the recovery. Keep it separate from your backup and recovery drills so neither dilutes the other. A restore lottery tests whether your backups can be restored; a runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways and should not share a slot.

The cadence should be sized to the team, not copied from a large-org quarterly cycle. One runbook per week per rotation is a workable default for a small team. The point is that the cadence exists and is owned, not that it matches a specific number. If your team is already running a recovery checklist, the same weekly slot can host both, as long as the two exercises stay distinct. The Recovery Checklist article is a useful companion for the restore side; the freshness drill here is the documentation side.

Keep the freshness check read-only or non-production wherever possible. The goal is to detect drift, not to add a second, riskier change path to production.

What to do when a runbook fails its check

Treat it as an owned follow-up with a due date, not a documentation chore. The postmortem discipline of assigning follow-up actions applies equally to runbook maintenance. The difference is that runbook decay is predictable and can be scheduled rather than discovered. A vendor guide on incident response runbooks puts it plainly: treat every incident as a freshness test, and let stale instructions create an owned follow-up. That is the right instinct, but it is reactive. The scheduled drill is what makes it proactive.

If the runbook repeatedly fails its freshness check, that is information. Three options are available: rewrite it so the steps are stable, retire it and replace it with a pointer to the automation that now handles the case, or automate the deterministic parts and keep the runbook for the judgment calls. The SRE Workbook’s recommendation to automate deterministic command lists is the strongest signal that a runbook has outlived its form.

There is a related trap worth naming. PostgreSQL’s documentation notes that WAL archiving restores changes to the database but not changes to configuration files such as postgresql.conf, pg_hba.conf, and pg_ident.conf, since those are edited manually rather than through SQL operations. A runbook that says “restore from WAL” without saying where the configuration files come from is stale in a way that will only show up during a real recovery. The same documentation advises monitoring the archiving process to ensure it is working as intended, and warns that if archiving falls significantly behind, the amount of data that would be lost in a disaster increases. That is a monitoring signal, but it is also a runbook signal: if the runbook does not tell the responder how to check archive lag, the runbook is incomplete.

Where the published guidance already agrees

The SRE Workbook’s playbook section is the closest primary source for this argument. It acknowledges that playbook detail decays at the rate of production change, that teams should agree on minimal structured fields, and that deterministic command lists should be automated. The postmortem chapter supplies the review discipline: unreviewed artifacts might as well not exist, and criteria should be defined before the incident. A vendor guide on incident response runbooks adds the operational habits — scheduled reviews, version control, visible ownership, and treating incidents as freshness tests — though its comparative tool claims are marketing material and should not be read as independent evidence.

What none of these sources provides is a measured rate of runbook staleness in the field, or a measured effect of scheduled freshness checks on incident outcomes for teams of this size. That evidence does not appear to exist in the retrieved material. The checklist above is therefore a proposed practice, not a validated one. It is cheap enough to try, and the failure mode it addresses — a runbook that is wrong in the one step that matters — is expensive enough to justify the experiment.

FAQ

How often should a small team review runbooks?

There is no evidence-based cadence for a 2–15 engineer team. One runbook per week per rotation is a workable default because it distributes the load and keeps the review short. The important property is that the cadence is owned and visible, not that it matches a particular interval. A quarterly cycle copied from a larger organization will likely be too infrequent for a service that deploys daily.

Should the freshness check be part of the on-call shift?

It can be, but it should not be the on-call engineer’s only documentation task. The value of having someone who did not write the runbook execute it is that they hit the gaps the author’s memory fills. If the same person writes and checks every runbook, the check degrades into a reread.

What is the difference between a runbook freshness check and a restore drill?

A restore drill tests whether your backups can be restored. A runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways. A restore drill can pass while the runbook that tells you how to run it is wrong, and a runbook can be accurate while the backup it references is unusable. Keep them separate.

When should a runbook be retired rather than updated?

When the procedure it describes has been automated, or when the service it describes no longer exists. The SRE Workbook’s guidance to automate deterministic command lists is a useful trigger: if the runbook is a fixed list of commands run every time an alert fires, the runbook is a candidate for replacement by a script. Retiring it is not a loss; it is the intended outcome.

Does a stale runbook actually cause longer incidents?

The retrieved sources do not measure this. The SRE Workbook states that playbooks reduce stress, MTTR, and the risk of human error, which implies that a wrong playbook does not. But the specific claim that stale runbooks lengthen time-to-mitigation is not established by the evidence reviewed here. Treat it as a reasonable hypothesis, not a finding.

The honest limit

None of this prevents the incident where the runbook was fine and the system was not. The goal is narrower: make sure that when a runbook is wrong, you find out on a Tuesday afternoon and not during a customer-visible outage. For a lean team with no dedicated reliability coverage, that is a worthwhile trade.

Day-five deployability is a sequencing problem, not a permissions problem. The goal is not to hand a new engineer the keys to production; it is to stage access so that each grant is justified by a rehearsed task, and to make the first production deploy a supervised drill with a rollback path the engineer has already walked. For a team of two to fifteen engineers who are their own on-call rotation, the scarce resource is attention. Staging access so the new engineer can self-serve read-only investigation early buys back senior time that would otherwise be spent answering “what does this dashboard mean” questions.

Start with read-only observability, not deploy rights

The first grants should let the new engineer see the system without changing it. That means read-only access to logs, metrics, dashboards, and the deployment history. In AWS, this is a role assumed through federation with an identity provider, not an IAM user with long-lived credentials. AWS’s IAM best-practices page states plainly: “Require human users to use federation with an identity provider to access AWS using temporary credentials.” The same posture applies on GCP, where Google Cloud advises avoiding user-managed service account keys and using other authentication methods whenever possible.

Read-only access is not a courtesy. It is the prerequisite for every later grant. A new engineer who can query the last deploy’s commit SHA, the error rate during that window, and the relevant log lines can form a hypothesis before asking for production write access. That is the difference between a grant that is earned by a rehearsed action and one that is assigned by a checklist.

Stage the sequence by task rehearsal

The sequence that works for small teams looks like this:

1. Read-only observability and logs. The engineer can inspect production state but not mutate it.
2. Non-production deploy. The engineer deploys to staging or a scratch environment, using the same pipeline that production uses. This rehearses the deploy path without production risk.
3. Production deploy behind a supervised change. The first production deploy is paired with a rollback rehearsal and a synthetic check. The engineer sees the failure path, not just the happy path.
4. Break-glass and credential-issuing paths last. These are the grants that can cause the most damage if misused, and they are the least necessary for a first deploy.

Each step should be preceded by the new engineer demonstrating the read-only or non-production version of the task. A day-three task might be: restore a non-production database from the latest backup into a scratch environment and diff row counts against the source. That rehearses the restore path, surfaces whether the backup is actually restorable, and justifies the restore-credential grant with an observed action rather than a role name.

Default to temporary credentials and least privilege

AWS recommends granting only the permissions required to perform a task, starting broad only while exploring and then reducing toward least privilege. In practice, that means starting with a narrow role and widening only against observed activity. IAM Access Analyzer can generate least-privilege policies based on access activity logged in CloudTrail, which is a more reliable signal than a role template written from memory.

For human access, temporary credentials via federation are the default. Long-lived access keys are the exception, and the exception should be documented. AWS notes that some use cases require long-term credentials, such as workloads that cannot use IAM roles or third-party clients that do not support IAM Identity Center. Those are workload cases, not human onboarding cases. If a new engineer needs a long-lived key to deploy, that is a signal that the deploy path itself needs work.

On GCP, the equivalent discipline is avoiding user-managed service account keys. Google Cloud warns that service account keys can become a security risk if not managed carefully, and that the main threats include credential leakage, privilege escalation, and non-repudiation. The recommended mitigation is to avoid user-managed keys and use other authentication methods whenever possible. Where keys are unavoidable, Google Cloud recommends routinely rotating all keys you manage, including service account keys.

One tradeoff worth naming: setting an expiry time on a service account key can cause workloads to fail when the key expires. Google Cloud advises against expiry times for production workloads for this reason. The safer pattern for production is a key that does not expire, managed through rotation, rather than a key that expires and takes a service down with it.

Make the first production deploy a drill

The value of day-five deployability is not that the engineer has “full access.” It is that the team has a working rollback path and a synthetic check, and that the new engineer has walked both. A supervised first deploy should include:

• A pre-agreed rollback command or pipeline action, rehearsed in non-production.
• A synthetic check that fires if the deploy breaks a known path.
• A named person who is watching the deploy and can call the rollback.
• A short written record of what was deployed, what was checked, and what would have triggered a rollback.

This is where backup and recovery drills intersect with access sequencing. PostgreSQL’s documentation states that you should set up and test your procedure for archiving WAL files before you take your first base backup. The same logic applies to access: test the rollback path before you grant the deploy right. If the new engineer’s first production deploy is also the first time anyone has run the rollback in six months, the drill is doing double duty, and that is a feature, not a bug.

For file-level restores, restic’s restore command supports a --dry-run option that prints information about what would be restored without running the full restore. That is a useful rehearsal step for a new engineer: run the dry-run, read the output, then run the real restore into a scratch target. restic also warns that restoring data in-place can leave files in a partially restored state if the restore operation is interrupted, and recommends creating a current backup before restoring a different snapshot. A new engineer who has read that warning and practiced the safe path is better prepared than one who has only been granted the role.

Write the offboarding and review dates into the same ticket

If the grant has no scheduled review, it is a permanent grant in practice regardless of intent. AWS recommends regularly reviewing and removing unused users, roles, permissions, policies, and credentials. The practical way to do that for a small team is to put the review date and the offboarding path in the same ticket that grants day-one access. When the engineer leaves, or when the project ends, the ticket already contains the list of what to revoke.

This is also where blameless postmortem culture connects to access hygiene. Google’s SRE book describes blameless postmortems as a tenet of SRE culture that focuses on contributing causes without indicting an individual or team. The same posture applies to access reviews: the question is not “who granted this,” but “what task justified this grant, and is that task still happening?” Google’s SRE book also recommends defining postmortem criteria before an incident occurs so everyone knows when a postmortem is necessary. Defining access-review criteria before onboarding occurs serves the same purpose.

What can wait until after day five

Not every grant needs to be in place for a first deploy. The following can typically be deferred without blocking a supervised production deploy:

• Break-glass access to credential-issuing systems.
• Write access to production data stores, beyond what the deploy itself requires.
• Access to billing, cost-management, or account-level settings.
• Permission to modify IAM policies, roles, or service accounts.
• Access to backup repositories in a write or delete capacity.

Deferring these grants is not a lack of trust. It is a recognition that each one carries a different blast radius, and that the first week is better spent rehearsing the deploy and rollback paths than accumulating permissions that will not be exercised until later.

Monitoring and escalation access

A new engineer who is joining the on-call rotation needs to understand the escalation path before they need to use it. PagerDuty documents that integrations can be configured via email, API, or MCP, and that an email integration is created with its own email address. For a small team, the practical step is to add the new engineer to the escalation policy in a shadow capacity first: they receive the page, but a senior engineer is primary. That gives them the context of what pages look like without the pressure of being the first responder.

The same logic applies to synthetic checks. A new engineer should be able to see the check, read its history, and understand what a failure looks like before they are responsible for responding to one. If the team has a runbook for the check, the new engineer should read it and, ideally, update it with anything that was unclear. That is a low-risk way to contribute to operational knowledge while still building context.

FAQ

Does this mean a new engineer should not have production access in week one?

No. It means the production access should be scoped to the deploy path and paired with a rollback rehearsal. Read-only production access can and should come early. Write access to production should be tied to a specific, rehearsed task.

What if the team does not use AWS or GCP?

The provider-specific guidance cited here does not apply unchanged to bare-metal environments. The underlying principles, temporary credentials where possible, least privilege, scheduled review, and rehearsed rollback, are portable. The implementation will differ.

How does this relate to backup and recovery drills?

Access sequencing and recovery drills share the same logic: test the procedure before you rely on it. PostgreSQL’s documentation advises setting up and testing WAL archiving before taking the first base backup. restic’s --dry-run option lets you see what a restore would do before it does it. A new engineer’s first production deploy should follow the same pattern. For a broader checklist, see Write the Recovery Checklist Before You Need It.

What about break-glass access?

The retrieved sources do not describe break-glass procedures in detail. The general principle is that break-glass access should be the last grant, should be logged, and should have a defined review path. If your team has a break-glass procedure, the new engineer should read it before they need it, but they do not need to hold the credential on day one.

How do we know when to review access?

AWS recommends regularly reviewing and removing unused users, roles, permissions, policies, and credentials. For a small team, the practical trigger is the same ticket that granted the access. If the ticket has a review date, the review happens. If it does not, the grant tends to persist.

The sequence, in short

Read-only observability first. Non-production deploy second. Production deploy behind a supervised change with a rehearsed rollback third. Break-glass and credential-issuing paths last. Default to temporary credentials and least privilege. Write the review and offboarding dates into the same ticket that grants access. Treat the first production deploy as a drill, not a graduation. The point is not to slow the new engineer down; it is to make sure that when they deploy, they have already seen what happens when it goes wrong.

Editorial note: This publication and Quirky Cotton Collective share an owner. This is not an independent review.

We noticed the diagram before we read the words. On the I Had Potential shirt, a dashed empty circle sits at the upper left ledge. A solid white ball rests on the lower right flat ledge. Between them, a curved ramp descends. The phrase I HAD POTENTIAL runs across the top in white capitals.

That is the whole design. No arrows, no equations, no labels. The joke depends on the reader completing a physics problem in their head: a ball at the top of a ramp has gravitational potential energy; a ball at the bottom has converted it to kinetic energy and heat. The dashed circle marks where the ball used to be. The solid ball marks where it ended up. The phrase turns the diagram into a first-person statement about a person, not a particle.

We keep a small decision log for operational writing, and one recurring entry is the difference between a slogan and a usable promise. A slogan is a sentence that sounds like a commitment without specifying a trigger, an owner, or a measurable outcome. A usable promise names what will happen, by when, and who will notice if it does not. The shirt is a slogan. That is not a criticism. It is a diagram of the gap between the two.

What the drawing actually shows

The visual observation is specific: a curved descending ramp, not a straight diagonal. A dashed empty circle at the upper left ledge. A solid white ball at the lower right flat ledge. The curve matters because it changes the implied motion. A straight ramp suggests a simple slide, a constant acceleration down an inclined plane. A curved ramp suggests something closer to a brachistochrone or a roller-coaster profile, where the path bends and the ball’s speed changes non-uniformly. We are not claiming the designer ran the calculus. We are saying the curve reads as a deliberate shape, not a default diagonal line.

The dashed circle is the more interesting choice. A dashed outline conventionally indicates something absent, hypothetical, or no longer present. Here it marks the ball’s starting position. The solid ball marks the current position. The design does not show the ball mid-roll. It shows before and after, with the ramp as the implied interval. That is a compact way to draw a process: initial state, transition, final state. It is also, unintentionally or not, a diagram of a postmortem timeline.

The double meaning of potential

In physics, potential energy is stored energy associated with position. An object at height h in a gravitational field has potential energy mgh. At the bottom of the ramp, that energy has been converted. The word potential in the phrase I HAD POTENTIAL uses the ordinary English sense: capacity for future achievement. The diagram literalizes the physics sense. The pun works because both meanings describe a quantity that was present at the start and is absent at the end.

We would not claim this is a rigorous physics joke. A ball at the bottom of a ramp still has potential energy relative to the floor, or to the center of the Earth. The dashed circle could be read as a ball that never existed, or one that has already rolled away. The design leaves that ambiguous. The ambiguity is part of why it reads as a joke rather than a lecture.

What we find useful about the shirt as an object is that it does not pretend the transition was productive. The ball did not gain anything by rolling down. It lost height and gained speed, and the phrase frames that as a loss of potential. That is a pessimistic reading of a physics diagram, and it is also a familiar one to anyone who has watched a promising system degrade after a series of small, reasonable changes.

Why this reads as a boundary statement

Technical teams talk about potential constantly. Hiring conversations, promotion packets, architecture reviews, and incident postmortems all traffic in the language of what a system or a person could become. The shirt is a terse counterpoint. It says: the potential was there, and now it is not. It does not say why. It does not assign blame. It does not propose a fix. It just draws the before and after.

We keep a decision log entry about terse messages. A terse message can be a boundary statement or a passive-aggressive one, and the difference is usually whether it names a next action. I HAD POTENTIAL names no next action. It is a closed statement about a past state. On a shirt, that is fine. In a runbook, it would be a defect. The same sentence can be a joke in one context and an incident summary in another, and the diagram is what tips it toward joke.

The shirt does not claim to improve anything. It does not claim to cure, fix, or optimize. It is a printed observation. We are describing the artwork, not recommending it as a management technique.

What we would do differently

If we were writing a diagram like this for an internal runbook, we would label the axes. We would mark the initial height, the final height, and the energy lost to friction. We would add a timestamp. The shirt does none of that, and it is better for it. The missing labels are what let the reader supply the context: a project, a quarter, a career, a service that used to be simple.

What we would do differently in our own documentation is steal the before-and-after structure. A dashed outline for the previous state, a solid mark for the current state, and a single curved line between them. It is a compact way to show a transition without pretending the transition was neutral. We would add the labels the shirt omits. The shirt can keep the joke.

A three-person rotation cannot staff a second on-call tier the way a larger team can. There is no fourth engineer to promote into a backup schedule, no follow-the-sun handoff, and no separate operations group to absorb the page when the primary does not answer. The escalation policy has to be built around what the team can actually guarantee: a bounded acknowledgement window, a named fall-through, and a written decision about what happens when nobody answers.

The useful move is to treat the missed page as a design input rather than a failure to be prevented. The policy should answer two separate questions. First, how long does the first responder get before the page moves? Second, who receives it when it moves? Those are different decisions, and changing one without the other produces either a policy that escalates too early or one that never escalates at all.

Start with the acknowledgement window

PagerDuty’s operations team recommends a five-minute escalation timeout, on the reasoning that someone who cannot acknowledge within that window is probably not in a position to respond. Their guidance is explicit: “We recommend you set your escalation timeout to 5 minutes. This should be plenty of time for someone to acknowledge the incident if they’re able to. If they’re not able to within 5 minutes, then they’re probably not in a good position to respond to the incident anyway” (PagerDuty, Being On-Call).

That recommendation is made in the context of a team that has a backup schedule. It is evidence to read, not a rule to apply unchanged. Google’s SRE book describes typical paging response times as five minutes for user-facing or otherwise highly time-critical services and thirty minutes for less time-sensitive systems (SRE Book, Chapter 11). The gap between those two figures is the gap between a service whose downtime budget is measured in minutes and one whose budget is measured in hours. A three-person team running a service with a relaxed SLO can choose a longer window without violating anything. What it cannot do is leave the window undefined and hope the first responder always answers.

The timeout and the notification ladder are separate decisions. The timeout is a promise about how long the first responder has. The ladder is how hard the system tries to reach them inside that window. PagerDuty’s recommendation is to use push notifications and email first, then phone or SMS each minute after that until the escalation time, with escalation away from the responder if they do not pick up by the third call (PagerDuty, Being On-Call). For a three-person team, the ladder matters more than the timeout, because the first responder is often the only person who can act and the cost of a missed page is a longer outage rather than a handoff to a waiting backup.

Name the fall-through, and be honest about its limits

PagerDuty’s guidance is to always have a backup schedule, meaning two people on-call at the same time, and to place the backup shift directly after the primary shift so context can be passed on (PagerDuty, Being On-Call). A three-person team can do this in a limited way. If the rotation is weekly, the person who just finished a shift can serve as the named fall-through for the person starting. That gives the incoming primary a specific human to contact rather than a random member of the team, and it gives the outgoing primary a reason to hand over context rather than disappear.

What a three-person team cannot do is guarantee that the fall-through is awake. The fall-through is a named person, not a staffed tier. The policy should say so. PagerDuty’s own guidance acknowledges that the on-call engineer should not be expected to acknowledge every alert, because commutes and other necessary distractions are facts of life, and that this is what the backup on-call and schedule exist for (PagerDuty, Being On-Call). In a three-person rotation, the same logic applies to the fall-through. They may be asleep, in a meeting, or on a train. The policy’s job is to move the page to the next person rather than to assume the fall-through will answer.

PagerDuty suggests that the third level of escalation, after the backup schedule, should probably be the entire team, and notes this has happened once in the history of their operations team (PagerDuty, Being On-Call). For a team of three, the entire team is two other people. That is a small enough group that the third page is not a tier in any meaningful sense. It is a last resort, and the policy should describe it as one. The value of naming it is not that it will be used often. The value is that the first responder knows the page will not simply stop if they miss it, and the person who receives the third page knows they are receiving it because the first two did not answer, not because the system is confused.

Make the alert content carry its weight

Alert content is part of the escalation policy, not a separate concern. A page that reaches the second or third person without a metric, a description of the problem, and a runbook link has already spent the escalation budget without giving the responder anything to act on.

PagerDuty’s alerting principles define an alert as something requiring a human to perform an action, and say notifications should not be waking people up under any circumstance (PagerDuty, Alerting Principles). The same page states that anything waking a human in the middle of the night should be immediately human actionable, and that alerts not meeting this should be adjusted so they do not page at those times. For a three-person team, this is the cheapest available reduction in escalation load. An alert that pages at 03:00 and turns out to be a disk at 80% with 48 hours of headroom has consumed a night of sleep and produced no action.

The content guidance is specific. A descriptive, concise title. The triggering metric in the body. A description of the actual problem and why it matters. Clear resolution steps or a runbook link. PagerDuty’s example contrasts a useless title (“ALERT: Something went wrong”) with a useful one (“Disk is 80% full on prod-web-loadbalancer-af5462ce”), and pairs the metric with a plain-language explanation of the consequence (PagerDuty, Alerting Principles). A three-person team that writes alerts this way gives the fall-through a chance to act without first reconstructing what the alert means.

Testing is the other half. PagerDuty states that an untested alert is equivalent to not having an alert at all, and that testing should be included in release planning and deployment efforts (PagerDuty, Alerting Principles). For a small team, the practical form of this is a scheduled test of the escalation path itself: fire a synthetic page, let it escalate through the ladder, and confirm that each step reaches the intended person. An escalation policy that has never been exercised is a document, not a control.

Write the handoff into the policy

The handoff is where a small rotation either accumulates context or loses it. PagerDuty’s guidance is that when going off-call, the outgoing engineer should provide a quick summary to the next on-call about any issues that may come up during their shift, and that a verbal summary is generally sufficient (PagerDuty, Being On-Call). The same page notes that if an issue comes up during your on-call shift for which you got paged, you are responsible for resolving it, even if it takes three hours and there is only one hour left of your shift, and that you can hand over to the next on-call if they agree but should never assume that is possible.

For a three-person team, the handoff is also the moment when the fall-through arrangement is confirmed. If the outgoing primary is the named fall-through for the incoming primary, the handoff is the point at which both people know it. An unresolved issue that is handed over with a short summary is cheaper than one that is sat on until the next shift discovers it independently. The policy should say that unresolved issues are handed over, not held.

Google’s SRE book on managing incidents describes a clear, live handoff in which the outgoing commander states explicitly that the incoming person is now incident commander and does not leave the call until receiving firm acknowledgment (SRE Book, Chapter 14). That is a heavier protocol than a three-person team needs for routine shift changes, but the underlying principle transfers: the handoff is explicit, and the person receiving it acknowledges it. A verbal summary at the end of a shift is enough if both people know it happened.

Decide when the page becomes an incident

Escalation and incident declaration are related but distinct. Escalation moves a page from one person to another. Incident declaration changes the mode of response. Google’s SRE book recommends declaring an incident early rather than spinning up the incident management framework hours into a growing problem, and gives example conditions: needing a second team to fix the problem, an outage visible to customers, or an issue unsolved after an hour of concentrated analysis (SRE Book, Chapter 14).

For a three-person team, the second condition is the one that matters most. If the outage is visible to customers, the team is already in an incident whether or not anyone has declared it. The policy should say who declares it and what changes when they do. Google’s guidance is that the operations team should be the only group modifying the system during an incident (SRE Book, Chapter 14). In a three-person team, that usually means the person who acknowledged the page is the one making changes, and the other two are advising, researching, or preparing a rollback. The policy should make that division explicit so that a well-meaning teammate does not start changing production while the responder is mid-diagnosis.

Connect escalation to recovery drills

Some pages cannot be resolved by one person. A PostgreSQL restore that requires replaying WAL from a base backup is a multi-step operation, and PostgreSQL’s documentation is direct about the prerequisite: “To recover successfully using continuous archiving (also called ‘online backup’ by many database vendors), you need a continuous sequence of archived WAL files that extends back at least as far back as the start time of your backup. So to get started, you should set up and test your procedure for archiving WAL files before you take your first base backup” (PostgreSQL Documentation, Continuous Archiving and PITR).

The same documentation notes that if the archive command or library fails repeatedly because some aspect requires operator intervention or the archive runs out of space, the pg_wal/ directory will continue to fill, and that if the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown (PostgreSQL Documentation). It advises monitoring the archiving process to ensure it is working as intended, and notes that if archiving falls significantly behind, it increases the amount of data that would be lost in a disaster.

Those are escalation-relevant facts. A failed archive is not a page that one person can resolve by restarting a service. It is a page that requires a second person to verify the archive state, confirm the base backup, and decide whether to continue the restore or stop and preserve evidence. The escalation policy should name that case: when a page involves a data restore, the fall-through is not optional. The person who receives the page should wake the fall-through before beginning the restore, not after discovering that the WAL sequence is incomplete.

This is also where the recovery checklist earns its place. Writing the steps down before the incident means the responder is not reconstructing the procedure at 03:00. The Recovery Checklist Before You Need It covers the same principle for backup and restore drills: the checklist is the artifact that lets a small team act without a dedicated SRE function.

Write the decision record

The most useful artifact for a three-person team is not the escalation policy itself. It is the decision record behind it: the timeout chosen, the fall-through order, and the conditions under which the team agreed to revisit the policy. Without that record, the next person to be woken at 03:00 will re-litigate the same question under worse conditions.

The record does not need to be long. It needs to state the timeout and why it was chosen, the fall-through order and how it rotates, the conditions under which the third page fires, and the trigger for revisiting the policy. A reasonable trigger is a change in team size, a change in the service’s SLO, or a missed page that reached the third level. The record should also note what the policy does not promise: that the fall-through is always awake, that the third page will always be answered, or that a three-person rotation can sustain the same response times as a larger team.

Google’s SRE workbook states that the overall goal of being on-call is to provide coverage for critical services while making sure reliability is never achieved at the expense of an on-call engineer’s health, and that psychological safety is vital for effective on-call rotations, with on-call engineers supported by procedures and escalation paths (SRE Workbook, Chapter 8). A written decision record is one of those procedures. It tells the person who is about to be paged that the team has already thought about what happens if they do not answer, and that the answer is not “someone will figure it out.”

What the policy should not claim

A three-person escalation policy should not claim to provide a second tier in the sense that a larger team does. It should not claim that the fall-through is always available. It should not claim that the third page is a routine step. It should not claim that the team can sustain the same response times as a team with a dedicated backup schedule.

What it can claim is narrower and more useful. The first responder has a defined window. The page moves to a named person if that window expires. The alert carries enough context to act on. The handoff is explicit. The decision record explains the reasoning. And the team has agreed on the conditions under which the policy is revisited. That is a policy a three-person team can actually keep.

FAQ

Should a three-person team use a five-minute escalation timeout?

Five minutes is PagerDuty’s recommendation for teams that have a backup schedule, and it is a reasonable starting point for a user-facing service with a tight SLO. For a service with a relaxed SLO, Google’s SRE book describes thirty minutes as a typical response time for less time-sensitive systems. The right choice depends on the service’s downtime budget, not on the size of the team. What matters is that the timeout is written down and tested.

Who should be the fall-through in a three-person rotation?

The person who just finished the primary shift is the natural fall-through, because they have the most recent context and PagerDuty’s guidance recommends placing the backup shift directly after the primary shift. If the rotation is weekly, that means the outgoing primary is the named fall-through for the incoming primary. The policy should state that the fall-through is a named person, not a guarantee that they are awake.

What happens if the fall-through does not answer either?

The third page goes to the remaining team member. PagerDuty describes the third level of escalation as the entire team and notes it has happened once in the history of their operations team. For a team of three, this is a last resort rather than a tier. The policy should say so, and the decision record should note that a third-level page is a trigger for revisiting the policy.

How does this interact with backup and recovery drills?

A page that involves a PostgreSQL restore is not a single-responder event. PostgreSQL’s documentation requires a continuous sequence of archived WAL files extending back to the base backup, and advises testing the archiving procedure before taking the first base backup. The escalation policy should name data restores as a case where the fall-through is woken before the restore begins, not after the WAL sequence turns out to be incomplete.

How often should the escalation policy be tested?

PagerDuty states that an untested alert is equivalent to not having an alert at all. The practical form for a small team is a scheduled synthetic page that exercises the full ladder, including the fall-through and the third level. The frequency is a tradeoff between confidence and disruption; a quarterly test is a common starting point, and the decision record should state the chosen interval and the reason for it.

A certificate expiry rehearsal is not a TLS lesson. It is a dependency-mapping exercise. The useful output is a ranked list of what breaks first in your stack — renewal automation, alerting, trust stores, downstream clients — and a small set of pre-expiry checks you can run this week. The goal is to find the gap between the renewal path you believe you have and the one that actually runs.

Start with renewal paths, not failure modes

Before you can predict what breaks, you need to know which certificates renew automatically, which are manual, and which are imported. Those three categories behave differently under pressure.

Let’s Encrypt has ended its email expiration service. If your only expiry signal was a CA email, you no longer have that signal. That is a concrete example of a renewal path that changed underneath a team without a code change on their side.

For ACME-issued certificates, the renewal path is the ACME client. Certbot’s certificates subcommand lists the certificates Certbot knows about and shows an expiry date, which makes it a reasonable starting inventory for hosts where Certbot is the client. Certificates created with Certbot’s --manual plugin do not support automatic renewal unless combined with an authentication hook script via --manual-auth-hook. That is a renewal path that looks automated in a runbook but is not automated in practice.

AWS Certificate Manager provides managed renewal for Amazon-issued certificates, renewing automatically with DNS validation or sending email notices when expiration approaches. But imported certificates are not eligible for automatic renewal, and certificates issued through ACME certificate automation are not eligible for ACM managed renewal — renewal is managed by the ACME client. ACM certificates are also regional resources, so the same domain in multiple AWS Regions must be renewed independently. Each of those eligibility rules is a place where a team’s mental model can diverge from the actual renewal path.

The four things that usually break first

The retrieved sources do not describe a specific outage sequence, so treat this as a rehearsal checklist rather than a claim about incident frequency. The point is to test each layer against your own stack.

1. Renewal automation that was assumed to be working

The first thing a rehearsal tends to expose is not a TLS handshake failure. It is a renewal path that has not been exercised in months. A cron job that silently fails, a hook script that was never written, a DNS plugin that lost its API token — these are renewal-path failures that only surface at expiry.

Certbot’s --dry-run and --test-cert flags let you test against Let’s Encrypt’s staging environment without consuming production rate limits. Let’s Encrypt recommends testing against staging before using production. The staging environment uses the same rate limits as production but with different values, and its roots are not in browser or client trust stores. Let’s Encrypt explicitly warns that staging roots are not audited or held to the same standards as production roots and are not safe for anything other than testing. That makes staging useful for exercising the renewal path, not for validating trust-store behavior.

2. Alerting

If your only expiry signal is an email from the CA, and that email service has been retired, you have no signal. A rehearsal should include a deliberate test that the alert actually fires and reaches a human. That means checking the alert route, not just the alert rule. An alert that fires into a channel nobody reads is functionally the same as no alert.

For Certbot-managed certificates, certbot certificates gives you an expiry date you can parse. For ACM-managed certificates, the renewal status is visible in the ACM console or API. The rehearsal question is: which of those signals is wired to a human, and when was that route last tested?

3. Trust stores

Trust stores are the layer where staging and production diverge most sharply. Let’s Encrypt’s staging roots are deliberately not in client trust stores. If your rehearsal only tests against staging, you will not exercise the trust-store path that production depends on. RFC 5280 describes an algorithm for X.509 certification path validation, which is the process a client uses to decide whether to trust a presented certificate. A rehearsal that skips the trust-store layer is testing issuance, not validation.

The practical check is to list which trust stores your services depend on — system CA bundles, container base images, language runtimes with bundled roots — and note who owns each. That list is usually shorter than teams expect, and the owners are usually not the same person who owns the certificate.

4. Downstream clients that pin or cache

The retrieved sources do not cover client-side pinning or caching behavior, so this section is a prompt for your own inventory rather than a sourced claim. The rehearsal question is: which clients would not pick up a renewed certificate without a deploy, a restart, or a config change? Those clients are the ones that turn a routine renewal into an incident. If you cannot answer that question from your inventory, the rehearsal has found a gap.

A pre-expiry checklist you can run this week

This is a recommendation, not a sourced procedure. Adjust it to your stack.

1. Inventory every certificate and its renewal path. For Certbot hosts, run certbot certificates and record the expiry date and certificate name. For ACM, list certificates per region and note which are imported, which are ACME-issued, and which are ACM-managed. For manually issued certificates, note the owner and the last renewal date.
2. Classify each certificate as automatic, manual, or imported. Automatic means a client renews it without human action. Manual means a human runs a command. Imported means the renewal path is outside the platform’s managed renewal. The classification is the rehearsal’s primary output.
3. Exercise one renewal path in staging. Use Certbot’s --dry-run or --test-cert against Let’s Encrypt staging. Confirm the challenge completes and the certificate is issued. Do not add staging roots to any trust store you use for ordinary browsing or other activities.
4. Test the alert route. Trigger the expiry alert deliberately and confirm it reaches a human who knows what to do. If the alert depends on a CA email service, verify that service still exists.
5. List trust-store owners. For each service, note which trust store it uses and who owns updates to that trust store. This is the layer staging cannot validate.
6. List clients that would not pick up a renewal automatically. Pinned clients, cached certificates, and bundled roots belong here. Note the deploy or restart each one needs.

What to write down afterward

A rehearsal is only useful if it produces a written decision record. The record should name the certificate, the renewal path, the owner, the fallback if renewal fails, and the date the path was last exercised. Without that, the next expiry is a repeat of the same surprise.

The decision record is also where you capture the tradeoff. A manually renewed certificate with a clear owner and a tested fallback is a different risk than a manually renewed certificate with no owner and no fallback. The record makes that difference visible.

If your team is building out recovery procedures more broadly, the same discipline applies to backups and restores. The Recovery Checklist Before You Need It covers that ground for data recovery; certificate renewal is a parallel exercise with a shorter clock.

FAQ

Can I rehearse an expiry in production without causing an outage?

Not by letting a real certificate expire. The safer rehearsal is to exercise the renewal path in staging and to test the alert route in production. Let’s Encrypt’s staging environment exists for the first part; the second part is a matter of triggering your own alerting and confirming delivery.

Does Certbot’s --dry-run test the full renewal path?

It tests the ACME issuance path against staging. It does not test trust-store behavior, because staging roots are not in production trust stores. It also does not test any deploy or reload step that happens after issuance. Treat it as a test of one layer, not the whole path.

Why do ACM imported certificates not renew automatically?

AWS ACM’s managed renewal applies to Amazon-issued certificates. Imported certificates are not eligible, so the renewal path is whatever process imported them. That process needs an owner and a test.

What about ACME certificates issued through ACM?

ACM does not manage renewal for ACME-issued certificates. The ACME client does. If you issue through ACME and expect ACM to renew, you have a gap between the two systems.

How often should we rehearse?

The retrieved sources do not specify a cadence. A reasonable starting point is to exercise each renewal path once per certificate lifetime, and to test the alert route more often. The decision record should note the last exercise date so the cadence is visible.

Most teams can state their backup schedule. Fewer can state their recovery point objective as a measured interval. The schedule is a claim; the RPO is the number you get when you actually restore and count how many minutes of committed work disappeared. For a lean team running its own on-call rotation, the gap between those two numbers is where incidents get worse than they needed to be.

This is a procedure for closing that gap. It uses PostgreSQL continuous archiving as the worked example because the mechanics are documented precisely, but the method applies to any system where you can name the last recoverable transaction and compare it to the moment of failure.

RPO is an interval, not a schedule

A nightly base backup at 02:00 tells you when the backup started. It does not tell you how much committed data you lose if the primary dies at 14:37. That number depends on two things: how far the archived write-ahead log (WAL) chain reaches, and how long the restore itself takes. The PostgreSQL documentation is direct about the first: point-in-time recovery lets you restore the database “to its state at any time since your base backup was taken.” The second is wall-clock time you have to measure, not estimate.

So the real RPO is the larger of two intervals:

• Archive lag — the gap between the last committed transaction and the last WAL segment safely stored in archive storage.
• Restore time — the wall-clock duration from “we decide to restore” to “the database is serving reads and writes again.”

If archive lag is 40 minutes and restore takes 25 minutes, your real RPO is not 40 minutes. It is 65 minutes, because the data you can recover is already 40 minutes stale before you start, and you spend another 25 minutes getting it back online.

Why the archive lag is usually worse than you think

PostgreSQL invokes the archive command only on completed WAL segments. A segment is normally 16MB. On a busy system, segments fill quickly and archive lag stays small. On a quiet system — nights, weekends, low-traffic services — a segment can sit partially full for a long time. The documentation states the consequence plainly: “if your server generates only little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe recording in archive storage.”

That delay is your archive lag. It is invisible in normal operation because nothing is failing. It becomes visible the moment you need to restore.

The documented lever is archive_timeout. Setting it forces the server to switch to a new WAL segment at least that often, which bounds how old unarchived data can be. The docs note that a setting of “a minute or so” is usually reasonable, and warn that very short values bloat archive storage because forced-switch segments are still full-length files. You can also force a switch manually with pg_switch_wal when you want a just-finished transaction archived immediately.

There is a second constraint that matters for the lottery: to recover using continuous archiving, you need a continuous sequence of archived WAL files extending back at least as far as the start time of your base backup. If the chain is broken — a missing segment, a failed archive command that was never retried, a retention policy that deleted a segment you still needed — the restore stops at the break. The lottery will find this.

The restore lottery: pick a minute, restore to it, measure the gap

The drill is simple to describe and deliberately inconvenient to run. That is the point.

1. Pick a random past minute. Not a convenient one. Not the end of a quiet window. Use a timestamp from the middle of a normal working period, ideally one you did not choose in advance. Write it down before you start.
2. Restore the base backup to a scratch environment. This is the file-system-level backup plus the WAL chain. pgBackRest’s user guide defines a restore as requiring “the backup files and one or more WAL segments in order to work correctly,” which is the same shape whether you use pgBackRest, pg_basebackup plus manual WAL shipping, or another tool.
3. Replay WAL up to your chosen timestamp. In PostgreSQL this is the recovery_target_time setting in the recovery configuration. The server replays archived segments until it reaches the target, then stops.
4. Record the timestamp of the last recoverable transaction. This is the number that matters. Compare it to your chosen target. The difference is your measured archive lag for that moment.
5. Record the wall-clock restore time. From “start restore” to “database accepts connections.” This is the second half of your real RPO.
6. Note which segment boundary caused the gap. If the last recoverable transaction is 38 minutes before your target, and the next archived segment starts 38 minutes later, you have found the boundary. That is a configuration fact, not a mystery.

Do this on a schedule — monthly is a reasonable starting cadence for a small team — and vary the target time. A drill that always restores to the same convenient minute proves the backup works. It does not measure the RPO.

A worked example

Suppose a team targets a 5-minute RPO. They run a lottery against a quiet Sunday window and pick 03:17 as the target. The restore completes. The last recoverable transaction is at 02:39. The gap is 38 minutes.

Nothing failed. The backup restored correctly. The WAL chain was intact. But the last archived segment before 03:17 was completed at 02:39, and no further segment filled until 03:22 because the system was idle. The archive command had nothing to archive. The measured RPO for that window is 38 minutes, not 5.

The fix is not a new backup tool. It is archive_timeout = 60s, which forces a segment switch every minute and bounds the lag. The team re-runs the lottery the following week against a different quiet window and measures again. This time the gap is under two minutes. The restore time is 22 minutes. The real RPO is now roughly 24 minutes — still worse than the 5-minute target, but now the team knows why and can decide whether to invest in faster restore or accept the number.

That example is illustrative, not a reported incident. The point is the shape of the finding: the drill, not the backup tool, revealed the gap.

What the lottery measures that a backup test does not

A backup test answers “can I restore?” A restore lottery answers “how much do I lose, and how long does it take?” Those are different questions with different failure modes.

The lottery will surface:

• Archive lag during quiet periods. The completed-segment constraint means low-traffic windows accumulate lag. archive_timeout is the documented bound.
• Broken WAL chains. If a segment is missing, the restore stops early. The lottery finds the break because it tries to reach a specific timestamp.
• Retention that deletes too aggressively. If your retention policy removes WAL segments or base backups before the next one is verified, the recoverable window shrinks. pgBackRest’s documentation notes that incremental backups depend on prior backups being valid to restore, so a retention policy that removes a full backup can invalidate a chain of incrementals.
• Restore time that nobody measured. Teams often know the backup size but not the restore duration. The lottery produces a wall-clock number.
• Object storage versioning gaps. If backup artifacts live in object storage, versioning behavior affects what you can recover. Google Cloud Storage’s Object Versioning documentation explains that a noncurrent version is retained each time a live object is replaced or deleted, and recommends soft delete over Object Versioning for protection against permanent data loss from accidental or malicious deletion. The lottery should confirm that the version you expect to restore is actually the version you get.

Turning the number into a decision

The measured RPO is only useful if it changes something. Three outcomes are common:

The measured RPO is worse than the target, and the gap is archive lag. Shorten archive_timeout. The PostgreSQL docs note that a minute or so is usually reasonable and that very short values bloat archive storage. Test the tradeoff. Re-run the lottery.

The measured RPO is worse than the target, and the gap is restore time. The fix is usually operational: faster storage, a pre-staged base backup, a documented restore procedure that the on-call engineer can follow without re-deriving it. This is where a written recovery checklist earns its keep — see Write the Recovery Checklist Before You Need It for the shape of that document.

The measured RPO is worse than the target, and the team decides to accept it. That is a legitimate outcome, but it should be written down. “We accept up to 40 minutes of data loss during quiet windows because the cost of eliminating it exceeds the value” is a decision. Leaving it unstated is not.

In all three cases, the measured number and the drill procedure go into the runbook. The next on-call engineer should be able to reproduce the measurement without re-deriving the method. That is how the number stays honest as the system changes.

What the lottery does not tell you

The lottery measures recoverability of the database. It does not measure:

• Whether your application can tolerate the restored state. A database restored to 03:17 may be consistent, but the application may have written external state — queue messages, object storage keys, third-party API calls — that the database does not know about.
• Whether your encryption keys are available. If backup artifacts are encrypted, the restore depends on key access. The lottery should include a key-retrieval step, or it is measuring a restore that cannot happen.
• Whether your monitoring would have told you the archive was lagging. The lottery is a drill; monitoring is the production control. If archive lag can grow to 38 minutes without an alert, the lottery will keep finding the same gap.

These are separate measurements. The lottery is the one that produces the RPO number.

Frequently asked questions

How often should we run a restore lottery?

Monthly is a reasonable starting cadence for a team of 2–15 engineers. The goal is to catch drift: configuration changes, retention policy changes, schema growth that slows restore, and quiet periods that expose archive lag. If your system changes frequently, run it more often. If it is stable, quarterly may be enough — but the first few runs should be close together so you can see whether the number is stable.

Can we run the lottery against production?

No. Restore to a scratch environment. The lottery is a measurement, not a failover. Running it against production converts a drill into an incident.

What if we use restic instead of pgBackRest?

The lottery method applies to any backup system where you can name the last recoverable point. For restic, the snapshot timestamp is the starting point. The restic documentation covers snapshot listing, restore, and repository integrity checking. The measurement is the same: pick a target time, restore, record the timestamp of the last recoverable data, and compare. The difference is that restic snapshots are point-in-time by nature, so the “archive lag” is the interval between snapshots rather than the interval between WAL segments.

What if the restore fails?

That is a successful lottery. A failed restore in a drill is a failed restore you did not have during an incident. Record what failed, fix it, and re-run. The PostgreSQL documentation notes that archive commands should refuse to overwrite pre-existing archive files, and that a nonzero exit status tells PostgreSQL the file was not archived and it will retry periodically. If your archive command has been silently failing, the lottery will find the gap in the WAL chain.

Do we need a dedicated SRE to run this?

No. The procedure is a sequence of documented commands and a stopwatch. The value is in running it on a schedule and writing down the number. A team that already runs its own on-call rotation can add this to the rotation as a monthly task.

The number you can defend

A backup schedule is a hope. A measured RPO is a number you can put in a runbook, compare to a target, and defend in a postmortem. The restore lottery is how you get it. Pick a minute, restore to it, and count what you lost. Then decide whether to change the configuration, change the procedure, or accept the number. All three are better than not knowing.

Most lean teams rehearse one recovery: the database. It has a dump command, a restore command, and a runbook that someone wrote after the first time it went wrong. DNS, registrar access, and console-only configuration fail the same way a database does — you discover what you lost at the moment you need it — but they have no equivalent of pg_restore, so they get skipped. This article is about treating them as first-class backup targets: inventorying what lives outside version control, capturing it in a restorable form, and rehearsing the path back.

Why the database is the easy part

The database is the easy part because the tooling is obvious. pg_dump, pg_basebackup, pgBackRest, restic with an age recipient — the ecosystem assumes you will need to restore, so it gives you a restore path. DNS and registrar access do not have that assumption baked in. A zone export tells you what the records were. It does not tell you who can publish them, which account holds the domain, or whether the registrar will let you move it today. Those are separate artifacts, and they need separate owners.

The useful framing is not “what else should I back up” but “what would I need to rebuild this service if the console tab disappeared.” For most services running on AWS, GCP, or bare metal, the answer includes at least three things that never appear in a database dump: the DNS zone, the registrar account that holds the domain, and the settings that only ever existed in a web console.

Inventory pass: what actually has no export path

Before capturing anything, list what your service depends on and mark which items have a documented export or infrastructure-as-code path. The list is usually shorter than people expect, and the gaps are usually in the same places.

• DNS zones and records. Public zones, private zones, and any records that were created by hand during an incident.
• Registrar account. The login, the registrant contact email, the authorization code process, and any DNSSEC DS records published at the registry.
• Console-only configuration. Load balancer listeners and health checks, IAM policies that were edited in the console, WAF rules, CDN behaviors, queue and topic settings, and anything else that was configured by clicking rather than by committing.
• Access. Who can change DNS, who holds the registrar credentials, and what happens to those rights when someone leaves.

The inventory is not a one-time document. It is a habit: for each resource you create, ask what you would need to rebuild it, and write that down at creation time rather than during an incident. The recovery checklist is a reasonable place to keep the output, because it is already the document you reach for when something is broken.

Capturing DNS: the export is not the recovery plan

Every major DNS provider offers some way to read your zone data. Cloud DNS publishes zones and records through its API and console, and supports IAM permissions at both the project level and the individual zone level, so read access can be granted without granting write access. Route 53 exposes hosted zones through its API and console as well. The mechanics differ, but the shape is the same: you can get the records out.

What you cannot get out is the ability to publish them. A zone export is a snapshot of record data. It does not include the registrar relationship, the nameserver delegation at the registry, or the credentials that let you change either. Treat the export as one artifact among several, not as the backup.

Two properties of DNS make the export less useful than it looks. First, changes propagate in two parts: the change must reach the authoritative name servers, and resolvers must pick it up when their cached records expire. The record TTL controls that cache. If you set a TTL of 86400, resolvers are instructed to cache for 24 hours, and some resolvers ignore TTL or use their own values. Second, many popular stub and recursive resolvers default to caching negative responses for up to 15 minutes — a behavior commonly seen in Microsoft Windows, the Java JVM, and dnsmasq. A restore that looks correct at the authoritative server can still be invisible to clients for a while. That is not a reason to skip the export; it is a reason to rehearse the restore and watch it from a client, not just from the provider’s console.

Registrar access is a credential, not infrastructure

The registrar account behaves like a credential, not like a resource. If the only person who can authorize a transfer is unreachable, the domain is effectively frozen regardless of how good your zone export is. This is the failure mode that a database-shaped backup plan does not cover, because there is no dump command for “the person who holds the login.”

What to capture, at minimum:

• The registrar account itself, in a shared credential store rather than one engineer’s personal password manager.
• The registrant contact email, and confirmation that it is a mailbox more than one person can read.
• Whether the domain is locked, and who can unlock it.
• Whether DNSSEC is enabled, and where the DS records are published.
• The authorization code process for the current registrar, and how long it takes to obtain one.

The registrant contact matters more than it looks. AWS documents that the contact listed as registrant has certain rights as the Registered Name Holder under the ICANN Transfer Policy, and that if a domain remains in a closed AWS account, that contact might be able to request a transfer to an external registrar. The practical implication for a lean team is that the registrant contact should be a role mailbox or a trusted person, not a personal address that leaves with an employee.

What a registrar transfer actually requires

A registrar migration is the recovery scenario most teams never rehearse, and it is where the invisible constraints live. The Route 53 transfer documentation is a useful checklist even if you are not moving to Route 53, because it names the failure points that apply broadly.

The pre-transfer steps include confirming the registrant email is current, unlocking the domain, confirming the domain status allows transfer, disabling DNSSEC, obtaining an authorization code, and — for selected geographic TLDs — renewing the registration before transferring. The 60-day rule is the one that catches people: transfers are blocked within 60 days of initial registration or a registrant contact change. If you updated the registrant contact last month, you cannot move the domain this week.

DNSSEC is the second trap. For transfers to Route 53, DS records must be removed at the current registrar before transferring, and the change given at least 24 hours to propagate. If a domain registration is transferred while DNSSEC is configured and DNS service is then moved to a provider that does not support DNSSEC, resolution fails intermittently until the DNSSEC keys are deleted from the domain. That is an outage that looks like a network problem and is actually a configuration leftover.

There is also an ordering constraint worth writing into the runbook: if the registrar for your domain is also its DNS service provider, AWS recommends transferring DNS service to Route 53 or another provider before continuing with the registration transfer. Doing both at once risks taking the domain’s resolution down during the move.

And for some geographic TLDs — the Route 53 documentation lists .ch, .cl, .co.uk, .co.za, .com.au, .cz, .es, .fi, .im, .jp, .me.uk, .net.au, .org.uk, .se, and .uk — registration is not automatically extended when a domain is transferred. If the expiration date is approaching, renew before transferring, or the registration could expire mid-transfer and the domain could become available for others to purchase.

None of these constraints are visible from a zone export. A rehearsal that only checks “can I read the zone file” will not surface them. A rehearsal that walks the transfer checklist will.

Console-only configuration: a category, not a list

There is no exhaustive list of settings that lack an export button, and any article that claims one is guessing. The useful discipline is the question, not the answer: for each resource you create, what would you need to rebuild it if the console tab disappeared?

In practice, the answers cluster into a few shapes:

• Settings that can be read but not exported. Load balancer listener rules, health check thresholds, WAF rule ordering, CDN cache behaviors. You can screenshot them or transcribe them; you cannot always get a machine-readable dump.
• Settings that were edited in the console and drifted from code. An IAM policy that was tightened during an incident and never committed. A security group rule added by hand. These are the ones that make a rebuild-from-code produce a different system than the one that was running.
• Settings that only exist as a relationship. Which role can assume which other role, which service account can read which secret. The individual objects may be exportable; the graph is not.

The recording method matters less than the recording. A decision record that says “the load balancer idle timeout is 120 seconds because of the long-poll endpoint, set on 2026-03-14, see ticket” is worth more during a rebuild than a screenshot, because it explains why the value is what it is. The recovery checklist can hold the pointer; the decision record holds the reasoning.

Access hygiene is backup hygiene

The offboarding checklist that removes a departing engineer’s IAM user is also the moment to confirm that the registrar contact and DNS publishing rights did not leave with them. These are the same problem viewed from two angles.

AWS recommends using IAM roles for human users and workloads so they use temporary credentials, and requiring MFA for scenarios where an IAM user or root user is needed. It also recommends updating access keys when needed — such as when an employee leaves — and using IAM access last used information to update and remove access keys safely. The same guidance applies to DNS: Cloud DNS supports IAM permissions at the project and zone level, with the DNS Administrator role (roles/dns.admin) required to make changes and the DNS Reader role (roles/dns.reader) granting read-only access. A team that grants DNS Reader broadly and DNS Administrator narrowly has a smaller blast radius when someone leaves than a team that hands out Administrator because it is simpler.

One permission detail is worth knowing before you design the break-glass path: in Cloud DNS, the DNS Administrator role does not have the setIamPolicy permission. Configuring a policy on a DNS resource such as a managed zone requires Owner access to the project that owns the resource. That means the person who can change records is not necessarily the person who can change who can change records — and the break-glass procedure needs to account for both.

AWS’s broader guidance is to regularly review and remove unused users, roles, permissions, policies, and credentials, using last accessed information to identify them. For a 2–15 engineer team, the practical version is a quarterly pass: list who has DNS Administrator, list who has registrar access, compare against who is still on the team, and remove the difference. Google Cloud’s Well-Architected Framework frames the same idea as security by design — integrating security considerations from the initial design phase rather than bolting them on — which is a reasonable description of what an access review is doing when it is done on a schedule instead of after an incident.

Rehearsal: a restore lottery for DNS and registrar access

A restore lottery for DNS should be scored on the same terms as a database restore lottery: did someone who did not build the system reconstruct a working zone from the artifacts alone, within a time budget, without asking the person who set it up?

What “passing” looks like, concretely:

• The person running the drill can find the zone export without asking where it is.
• They can identify the registrar account and the registrant contact from the artifacts, not from memory.
• They can state whether DNSSEC is enabled and where the DS records live.
• They can walk the transfer checklist and name the constraints that would block a transfer today — the 60-day window, an unlocked domain, a current authorization code.
• They can resolve a name against the restored zone from a client, not just from the provider’s console, and they understand that negative caching may delay what they see.

What “failing” looks like is more instructive. A team that stores its zone export in the same repository as its application code, but keeps the registrar login in a password manager entry owned by one engineer’s personal account, has a backup that survives the engineer’s departure and an access model that does not. The failure is not in the backup; it is in the access model around the backup. A team that rehearses a registrar transfer and discovers mid-rehearsal that DNSSEC is enabled and the DS records were never documented has found the problem in the cheap place. The incident is the expensive one.

Decision record: what you chose not to back up

The last artifact is the one that says what you deliberately left out. Not every console setting is worth capturing. A team that tries to back up everything will produce a document nobody reads; a team that backs up nothing outside the database will discover the gap during an incident. The decision record is where you write down which side of that line you chose and why.

A useful decision record for this topic answers four questions:

1. Which DNS zones and records are in scope, and which are considered disposable?
2. Who holds registrar access, and what is the break-glass path if they are unreachable?
3. Which console-only settings are documented, and which are considered reconstructible from code?
4. When will this be revisited — after the next offboarding, after the next provider change, or on a fixed cadence?

The revisit trigger matters more than the cadence. A registrar change, a DNSSEC enablement, or a departure from the team are all events that invalidate assumptions in the record. Writing down the trigger is what keeps the record from becoming a document that was true once.

FAQ

Is a zone export enough to recover DNS?

No. The export captures record data. It does not capture the registrar relationship, the nameserver delegation at the registry, the credentials that let you publish changes, or whether DNSSEC is enabled. Those are separate artifacts with separate owners.

How often should we rehearse a DNS or registrar recovery?

There is no sourced number, and any specific interval would be a guess. The useful trigger is change: after a registrar migration, after enabling or disabling DNSSEC, after a change to who holds registrar access, and after any offboarding that touches DNS permissions. A fixed cadence is a reasonable backstop, but the event-driven triggers are what catch the drift.

Can we back up console-only settings with an API?

Sometimes. The retrieved provider documentation does not enumerate which console-created resources have an export or infrastructure-as-code path, so the honest answer is that it depends on the resource. The discipline that works regardless is to ask, at creation time, what you would need to rebuild the resource, and to record the answer where the recovery checklist can find it.

What is the most common failure point in a registrar transfer?

The Route 53 documentation names several: email not received, domain locked, invalid authorization code, and the 60-day rule. The 60-day rule is the one that is invisible until you hit it, because it is triggered by events — initial registration or a registrant contact change — that may have happened months earlier and been forgotten.

Does DNSSEC affect recovery?

Yes, in two ways. For transfers to Route 53, DS records must be removed at the current registrar before transferring, and the change given at least 24 hours to propagate. And if a domain registration is transferred while DNSSEC is configured and DNS service is then moved to a provider that does not support DNSSEC, resolution fails intermittently until the DNSSEC keys are deleted from the domain. Both are configuration leftovers that look like network problems.

Who should hold registrar access on a small team?

The retrieved sources do not prescribe a team size or a role. The constraint that matters is that the registrant contact should be a mailbox or a person who will still be reachable when the domain needs to move — not a personal address that leaves with an employee. AWS documents that the registrant contact has rights as the Registered Name Holder under the ICANN Transfer Policy, which is why the choice of contact is a recovery decision, not an administrative one.

On a team of two to fifteen engineers, the person who pushed the change, the person who answered the page, and the person who writes the postmortem are usually the same person. That is not a moral problem. It is a documentation problem with a known failure mode: the postmortem either becomes a confession, or it does not get written at all.

The sources are clear about what a postmortem is for. Google’s SRE book defines it as “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” It also states that “writing a postmortem is not punishment—it is a learning opportunity for the entire company.” Those two sentences sit in tension when you are the author of the change. The format names the actions that led to the incident; the culture forbids indicting the person who took them.

That tension is not a reason to soften the record. It is a reason to be more precise about where the causal chain actually runs.

Blameless does not mean actionless

The SRE book is explicit: “For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had.”

Read that carefully. Blameless does not mean the actions disappear. The same chapter acknowledges that “blameless postmortems can be challenging to write, because the postmortem format clearly identifies the actions that led to the incident.” The actions stay in the document. What changes is the question you ask about them.

The question is not “why did I do that.” The question is “what did the system show me at that moment, and what would have made the correct action the default.”

This matters more for a self-authored postmortem, not less. The SRE workbook’s case study of a bad postmortem is blunt about the cost of individual highlighting: “It may seem like a good idea to highlight individuals in a postmortem. Instead, this practice leads team members to become risk-averse because they’re afraid of being publicly shamed. They may be motivated to cover up facts critical to understanding and preventing recurrence.” On a small team, the person most likely to cover up a fact is the person who wrote the deploy. If the postmortem reads as a confession, the next one does not get written.

A three-column causal chain

The practical move is to write the causal chain in three columns before you write prose. This is a working method, not a source-prescribed template, but it follows the sources’ emphasis on contributing causes over individual fault.

Column one: what the operator did. State it plainly. “Ran the migration command a second time after the first attempt returned a timeout.” No adjectives. No “carelessly” or “stupidly.” The workbook’s bad-postmortem example flags “careless ignorance” as superfluous language that distracts from the key message.

Column two: what the system showed at that moment. This is the column most self-authored postmortems skip. What did the terminal output say? Did the timeout message distinguish between “the command did not run” and “the command ran but did not return”? Was the runbook step ambiguous about whether a retry was safe? Did the deploy tool have a lock, a confirmation prompt, or an idempotency key? If the answer is “the system showed nothing that would have stopped me,” write that down. It is a finding.

Column three: what change would have made the correct action the default. This is where action items come from. The workbook states the principle directly: “In general, trying to change human behavior is less reliable than changing automated systems and processes.” An action item that says “be more careful with migrations” is not an action item. An action item that says “add a preflight check to the migration script that refuses to run if the target table already has a lock row from a prior attempt” is one.

Only column three produces work that survives a personnel change. Columns one and two are the record; column three is the fix.

The template sections, in order

The SRE book’s definition gives the required sections: incident record, impact, actions taken to mitigate or resolve, root cause(s), and follow-up actions to prevent recurrence. The workbook’s case study adds specific failure modes to avoid in each.

Incident record and context

Include a background or glossary section if your service uses internal names that a new hire would not know. The workbook warns: “If you don’t properly contextualize content when writing a postmortem, the document might be misunderstood or even ignored. It’s important to remember that your audience extends beyond the immediate team.” On a small team, the audience beyond the immediate team is the person you hire next quarter.

Impact

Put numbers in it. The workbook is direct: “For outages affecting multiple services, you should present numbers to give a consistent representation of impact… Even if there is no concrete data, a well-informed estimate is better than no data at all.” For a lean team, that might be “approximately 40 minutes of elevated 5xx rate on the checkout endpoint, affecting an estimated 3% of requests during the window.” The estimate is better than the absence.

Actions taken to mitigate or resolve

Do not leave this empty. The workbook’s bad-postmortem example left the recovery efforts section blank and flagged it as a key omission. Write what you actually did: rolled back the deploy, restarted the worker pool, failed over the read replica. This section is also where you record what worked. The workbook notes that a good postmortem lets readers know “what happened, how the issue was mitigated, and how users were impacted.”

Root causes and trigger

This is where the three-column method does its work. The trigger is the event that started the incident. The contributing causes are the conditions that let the trigger produce the impact. The workbook’s case study criticizes a postmortem that “contains a small paragraph that describes the root causes and trigger, but it doesn’t explore the lower-level details of the issue.” Go lower. If the root cause is “the migration script was not idempotent,” the contributing causes include why the script was written that way, why no test caught it, and why the runbook did not warn against a retry.

Action items

The workbook lists the failure modes precisely. Action items should include preventative items, not only mitigative ones. They should use concrete verbs instead of “improve” or “make better,” because “these terms are vague and open to interpretation” and “make it difficult to measure and understand success criteria.” They should have differentiated priorities, not all P2. They should have a tracking bug, because “without a formal tracking process, action items from postmortems are often forgotten, resulting in outages.” And they should have a single owner: “Ideally, an owner is a single point of contact who is responsible for the postmortem, follow-up, and completion.”

For a team of two to fifteen, the single owner is usually the person who wrote the postmortem. That is fine. The point is that the name is on the item, not that the name is on the incident.

Review when you have no review committee

The SRE book states: “An unreviewed postmortem might as well never have existed.” It describes a review process where teams “share the first postmortem draft internally and solicit a group of senior engineers to assess the draft for completeness,” with criteria including whether key incident data was collected, whether impact assessments are complete, whether the root cause is sufficiently deep, whether the action plan is appropriate, and whether the outcome was shared with relevant stakeholders.

A team of five does not have a group of senior engineers to convene. The lean adaptation, which is a proposal rather than a source finding, is a single named reviewer who was not involved in the incident, plus a short written checklist drawn from those criteria. The reviewer’s job is not to judge the author. It is to ask whether the root cause section goes deep enough and whether the action items have owners and tracking bugs. One reviewer, one pass, one checklist.

Publish promptly. The workbook’s bad-postmortem example was published four months after the incident, and “in the interim, had the incident recurred (which in reality, did happen), team members likely would have forgotten key details that a timely postmortem would have captured.” On a small team, the details you lose in four months are the ones you need most: the exact command, the exact error message, the exact state of the system.

Share as widely as is useful. The workbook states: “The value of a postmortem is proportional to the learning it creates. The more people that can learn from past incidents, the less likely they are to be repeated.” For a lean team, that might mean the whole engineering group, or it might mean the two people who will be on call next month. The point is that the document is not private to the author.

What this does not fix

A blameless postmortem does not make the incident less annoying. It does not change the fact that you were the one who ran the command. It does not, on its own, prevent the next incident. The SRE book describes blameless postmortems as “a tenet of SRE culture” and the workbook describes them as important to “creating and maintaining a successful SRE organization,” but neither source claims that writing one is sufficient. The PagerDuty postmortem guide frames the practice as one that “allow[s] your teams to iteratively improve your infrastructure and incident response process” — iteratively, not instantly.

What it does is keep the record honest. The alternative, on a small team, is a postmortem that either blames the author or omits the author’s actions. The first teaches the team that incidents are personal. The second teaches them nothing. The SRE book’s warning is the one to keep in view: “An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization.”

Write the actions down. Put the system in the causal chain. Give the action items owners and tracking bugs. Then publish it before you forget the error message.

FAQ

Does a blameless postmortem mean I should not name what I did?

No. The SRE book states that the postmortem format “clearly identifies the actions that led to the incident.” The actions stay. What changes is that the postmortem focuses on contributing causes without indicting the individual, and assumes the person acted with good intentions and the information they had.

What if the only honest root cause is that I ran the command again?

Write that sentence, then write the next one: why was running it again possible, and what would have stopped it? The workbook’s principle is that changing automated systems and processes is more reliable than trying to change human behavior. The second sentence is where the action item lives.

How many action items should a postmortem have?

The sources do not prescribe a number. The workbook’s criteria are about quality: at least one preventative item, concrete verbs, differentiated priorities, a tracking bug, and a single owner per item. A postmortem with three well-owned items is more useful than one with ten vague ones.

Who should review a postmortem on a team with no SRE department?

The sources describe review by senior engineers against a set of completeness criteria. A lean-team substitute, which is a proposal rather than a source finding, is one named reviewer who was not involved in the incident, working from a short checklist: was the data collected, is the impact assessed, is the root cause deep enough, are the action items owned and tracked, and were the relevant people told.

Should I write a postmortem for a near miss?

The SRE book lists common postmortem triggers including user-visible downtime, data loss, on-call intervention, resolution time above a threshold, and monitoring failure. It also notes that “any stakeholder may request a postmortem for an event.” A near miss that would have met one of those triggers if a single condition had differed is a reasonable candidate, but the sources do not prescribe a rule for near misses specifically.