Gray Haven Lab

The first thing we notice is the triangle. A yellow caution triangle, the kind that usually means wet floor or high voltage, sits above three lines of block lettering: NOT FRIENDLY, DO NOT TOUCH. Below the words, a hand is crossed out — a simple pictogram of a palm with a diagonal slash through it. The whole composition is borrowed from industrial signage, the visual grammar of hazard warnings, and it is doing something slightly different here. It is not warning you about a machine. It is warning you about a person.

That shift — from equipment to human — is the entire joke, and also the entire argument. The shirt is not a slogan in the usual sense. A slogan makes a claim about the wearer’s mood or taste. This is closer to a boundary marker: a terse, legible statement of access policy, printed in the one visual language most people already know how to read without thinking. We keep coming back to that distinction because it matters for how the object works in practice. A shirt that says not friendly in plain text is a joke. A shirt that says it inside a caution triangle, with a crossed-out hand, is a joke that has been formatted as an instruction.

What the sign language actually does

Industrial signage is designed to be parsed at a glance, by people who are not paying attention, under conditions where a mistake has consequences. The caution triangle, the crossed-out hand, the all-caps imperative — these are not decorative choices. They are a compressed protocol. When you put that protocol on a t-shirt, you are borrowing its authority. The wearer is not just saying leave me alone; they are saying it in the register of a system that expects to be obeyed.

We think that is why the design reads as more than a novelty. The crossed-out hand is the key element. It is not a hand waving, not a handshake, not a hand held up in a stop gesture. It is a hand with a slash through it — the universal symbol for this action is prohibited. The shirt is not describing a personality trait. It is prohibiting a behavior. That is a meaningful difference, and it is the difference between a shirt you wear to be funny and a shirt you wear to be left alone.

We should be careful here. We are reading the artwork, not making a claim about the wearer’s actual social preferences. The design is a visual argument, not a diagnostic tool. But the argument it makes is coherent: it takes the grammar of a hazard sign and applies it to the hazard of unwanted interaction. Whether that lands as funny, rude, or simply accurate depends entirely on the room.

The difference between a slogan and a usable promise

In our day jobs we spend a lot of time thinking about the difference between a statement and a procedure. A runbook that says restore from backup is a slogan. A runbook that says run this command, check this output, escalate to this person if it fails is a usable promise. The gap between those two things is where most operational failures live.

The Not Friendly Do Not Touch design sits on the usable-promise side of that gap, at least visually. It does not ask the reader to infer the wearer’s boundaries from context. It states them in the most legible format available. The caution triangle is a trigger. The crossed-out hand is the action. The lettering is the policy. It is, in its own small way, a well-designed interface: the reader knows exactly what is expected of them within about half a second of looking at it.

That is not nothing. Most social signals are ambiguous by design. This one is not. Whether you find that refreshing or abrasive probably depends on how much of your day you spend explaining boundaries that you wish were already obvious.

Why the warning-sign format is the right container

We have written before about shirts that use diagram language — the ramp-and-ball physics pun, the composition notebook surface — and the common thread is that the visual format does most of the work. A joke printed in plain text has to be read. A joke printed as a diagram has to be recognized. Recognition is faster and, for a certain kind of wearer, more satisfying.

The warning-sign format is particularly well suited to this message because it carries an implicit this is not personal clause. A hazard sign is not angry at you. It is not judging you. It is simply informing you that a condition exists and that a behavior is prohibited. When the wearer puts that format on, they are borrowing that same impersonal tone. The shirt is not saying I dislike you. It is saying this area is restricted. That is a much easier thing to wear in public, and a much easier thing to laugh at, because it removes the sting of rejection from the message.

We think that is the smartest thing about the design. It takes a boundary that could read as hostile and formats it as infrastructure. The crossed-out hand is not a rejection of a specific person. It is a policy. Policies are easier to accept than insults.

What the shirt is not

It is not a claim about introversion as a clinical category, and we would not read it as one. The retail title uses the word introvert, but the artwork itself does not. The artwork says not friendly and do not touch, which are behavioral instructions, not personality labels. We mention this because it is easy to collapse the two, and the collapse does the design a disservice. The shirt is funnier and more useful as a boundary marker than as a personality badge.

It is also not a substitute for actually telling people what you need. A shirt is a low-stakes signal, not a substitute for a conversation. We would not want to overstate its power. But as a low-stakes signal, it is unusually well constructed: legible, consistent, and formatted in a visual language that most people already know how to read.

Where it fits

We keep a small decision log for objects like this — things that are not tools but that shape how a team interacts. A shirt is not a runbook. But the question it raises is the same one we ask about runbooks: does the reader know what to do next? The Not Friendly Do Not Touch design answers that question clearly. The reader knows exactly what to do next. They know not to touch. They know the wearer is not friendly. The interface is complete.

If you want to look at the listing, it is here: Not Friendly Do Not Touch Funny Warning Sign Introvert. We are not going to pretend we tested it. We read the artwork, and the artwork is doing something specific and coherent. That is enough for us to recommend it as a well-designed object, which is a different thing from a recommendation to buy it.

What we’d do differently

If we were designing a shirt like this, we would probably be tempted to add more text — a second line, a clarification, a joke that undercuts the boundary. We would be wrong. The strength of this design is its restraint. The caution triangle, the crossed-out hand, the three words. Nothing else. The temptation to explain is the enemy of a usable signal, and this shirt resists it. We would do well to remember that the next time we are writing a runbook and reach for a fourth paragraph of context. Sometimes the crossed-out hand is enough.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

On a dark shirt, centered, in very small lowercase letters, the word meh. appears. No animal illustration. No pride variant. No diagram, no arrow, no punchline in a second line. The period is part of the mark. The scale is the argument: the word is set small enough that a reader has to be close to parse it, which means the shirt does not perform for a room. It performs for one person at conversational distance, if it performs at all.

That is a different object from the slogan shirt as it usually works. Most slogan shirts are legible across a parking lot. They are designed to be read by strangers, which makes them a kind of broadcast. This one is designed to be read by someone already close enough to notice. The difference matters for anyone who has spent time thinking about how teams communicate under stress, because it is the same difference between a status page and a runbook.

We keep a small decision log for our own systems, and one recurring entry is about audience. A status page is written for people who are not us: customers, support, whoever is watching the graph. A runbook is written for the person who will be paged at 3 a.m. and needs the next command, not a narrative. Both are text. Both are terse. But they are calibrated to different distances, and confusing them produces the specific failure where a status page reads like an internal note or a runbook reads like a press release.

The meh. shirt is calibrated to the close distance. It is not a status page. It is not trying to inform a crowd that something is wrong. It is closer to a runbook entry that says only what the operator needs and nothing more: a single word, a period, done. The period is doing real work. Without it, meh is a sound. With it, meh. is a verdict. The punctuation turns an interjection into a statement, and the small type keeps the statement from becoming a speech.

I want to be careful about what this observation supports. It supports a reading of the artwork. It does not support a claim that the shirt improves anything, communicates a policy, or represents a movement. It is a garment with a word on it. The interesting question is not whether it is deep. The interesting question is what kind of message survives being set this small, and why a designer would choose that constraint.

Small type is a filter. It selects for proximity and attention. In a work context, the messages that survive that filter tend to be the ones that do not need volume: a boundary, a handoff, a single line in a channel that says the thing and stops. The messages that fail the filter are the ones that need an audience to complete them. A joke that requires a laugh track is not the same object as a joke that lands in a quiet room. The meh. shirt is built for the quiet room.

There is a useful distinction here between a slogan and a usable promise. A usable promise tells you what will happen next. It has an owner, a trigger, and a condition. A slogan tells you what someone feels or claims. The meh. shirt is a slogan, and I think that is the correct category for it. It does not promise uptime, recovery, or anything operational. It does not pretend to be a runbook. It is a mood rendered as typography, and the restraint is what keeps it honest. A shirt that tried to be a usable promise would need more words, and more words would make it worse.

This is where the object connects to a habit I think technical teams should steal from designers: decide the distance before you decide the words. If the message is for the person already in the room, you can be terse. If it is for the person who has not arrived yet, you need context. Most communication failures I have watched are distance failures, not tone failures. Someone writes a close-distance message and broadcasts it, or writes a broadcast message and wonders why the on-call engineer ignores it. The fix is rarely more empathy. It is usually a smaller audience and a shorter sentence.

The shirt is available as a standard t-shirt through the listing at https://www.amazon.com/dp/B0FXC6KKSR. I am describing the artwork as observed in the native garment preview, not a wearing experience. I have not tested the fabric, and I am not going to pretend that a shirt is a reliability practice. It is an object with a word on it, and the word is set small on purpose.

What I keep returning to is the period. In a runbook, the period is the end of a step. It means the command is complete and the next thing can happen. In the shirt, the period does something similar: it closes the word so it does not trail off into a question or a shrug. meh. is not ambivalence. It is a decision to stop. That is a small, repeatable move, and it is one I would like to see more of in incident communication: say the thing, end the sentence, wait for the next signal.

What we’d do differently

If we were writing the listing copy, we would resist the urge to explain the joke. The artwork is already doing the work, and explanation would only add distance. We would also avoid claiming the shirt is for a particular personality type, because the design does not establish that. The observation is the type: small, centered, lowercase, dark ground, period included. Everything else is the reader’s business.

A bus factor audit is not a documentation project. It is a mapping exercise. The deliverable is a per-service list of people who can actually restart the service, plus the evidence behind each name. If the list has names but no evidence, you have a hypothesis, not a map.

For a team of 2–15 engineers who are their own on-call rotation, the audit has one job: find the services where the answer to “who can restart this?” is a single person, a stale runbook, or a name that has never done it under pressure. Then turn those gaps into drills, shadowing, and runbook work before the next incident finds them for you.

Why this is a mapping exercise, not a documentation project

Documentation projects tend to produce artifacts that describe how a service works. A bus factor audit produces a different artifact: a table that says who can perform a specific recovery action and how you know. The distinction matters because a runbook can be complete and still leave you with a bus factor of one. The runbook is not the capability. The person who has executed the runbook is the capability.

PagerDuty’s incident response guidance notes that knowledge of how to handle incidents is often lost when a team member leaves, which is why a documented process becomes more crucial over time. That is the failure mode the audit is designed to surface before it becomes an incident. The audit does not replace the documentation; it tells you where the documentation is the only thing standing between you and an unowned service.

NIST SP 800-34 Rev. 1 frames contingency planning as a process of evaluating information systems and operations to determine requirements and priorities. A bus factor audit is a narrow, repeatable version of that evaluation, scoped to restart and recovery actions rather than the full contingency plan.

Define the unit of audit: service, restart action, evidence

The unit of work is not the service as a whole. It is the restart action. A single service may have several restart paths, and different people may be able to perform each one:

• Process restart on the primary host or container.
• Failover to a standby or secondary region.
• Restore from backup (database, object store, configuration).
• Rebuild from infrastructure-as-code when the host is gone.
• Break-glass access path when normal credentials are unavailable.

For each path, record the evidence type. Three levels are enough for most lean teams:

• Has done it in an incident. The strongest evidence. The person executed the action under real pressure, with real consequences.
• Has done it in a drill. Strong evidence. The person executed the action in a controlled setting, ideally with a time constraint and a defined success condition.
• Has read the runbook. Weak evidence. This is a starting point, not a qualification.

Google’s SRE workbook describes a pre-on-call checklist that includes administering production jobs, rolling back a bad software push, bringing up additional serving capacity, and using monitoring systems. That checklist is a useful model for the evidence bar: the question is not “have you read about this?” but “have you practiced this?”

Build the map

One row per service. Columns for primary, backup, evidence type, last verified date, and known gaps. Keep it in a place the team already looks at, not a separate tool that will be forgotten.

The table is deliberately small. For a 2–15 engineer team, the audit should be completable in a few sessions. If it is taking longer, the scope is too broad. Start with the services that have the highest blast radius and the fewest people.

Use the map to find single points of failure

Three patterns are worth flagging explicitly:

• One name, no backup. The service has a primary but no second person who has executed the restart path. This is the classic bus factor of one.
• Names but no evidence. The service has a primary and a backup, but both are at “has read the runbook.” This is a bus factor of one with extra steps.
• The runbook is the only artifact. No one on the current team has executed the restart path, and the runbook was written by someone who has left. This is the highest-risk category because the runbook may be stale in ways no one can see.

Google’s SRE workbook notes that playbook details go out of date at the same rate as production environment changes, and that for daily releases, playbooks might need an update on any given day. A runbook that has not been executed by a current team member is a runbook whose staleness is unknown.

Turn gaps into drills

The audit’s output should feed directly into drills and shadowing, not into a separate documentation backlog. The goal is to move people from “has read the runbook” to “has done it in a drill.”

Google’s SRE workbook describes the Wheel of Misfortune as a training exercise where teams role-play recent incidents to practice debugging production problems. A bus factor drill is a narrower version: pick a service with a gap, pick a person who has not executed the restart path, and run the restart in a controlled setting. The success condition is not “the service came back” but “the person can describe what they did and why.”

For restore paths, a restore lottery is a useful format. One person is selected at random, given the backup and the runbook, and asked to restore to a point in time. The exercise surfaces both the person’s capability and the runbook’s accuracy. If the restore fails because the runbook is wrong, that is a finding, not a failure.

For services with no second person, shadowing is the first step. The primary executes the restart path while the backup observes and asks questions. The next step is the reverse: the backup executes while the primary observes. Only after that does the backup count as evidence.

Keep the map alive

A bus factor map that is not reviewed becomes a stale spreadsheet. Tie the review to artifacts the team already uses:

• On-call handoffs. When the rotation changes, check whether the new on-call has executed the restart paths for the services they are now responsible for.
• Playbook updates. When a runbook changes, mark the corresponding map row as unverified until someone executes the new path.
• Postmortem action items. When a postmortem identifies a recovery gap, add it to the map with a review date.

The Google SRE book states that an unreviewed postmortem might as well never have existed, and that regular review sessions are encouraged to ensure each completed draft is reviewed. The same logic applies to the bus factor map: an unreviewed map is a map you cannot trust.

Google’s SRE book lists common postmortem triggers including user-visible downtime or degradation beyond a certain threshold, data loss of any kind, on-call engineer intervention, a resolution time above some threshold, and a monitoring failure. Any of these triggers is also a signal to re-check the map for the affected service.

What to do when the map shows a gap you cannot close this quarter

Not every gap can be closed immediately. A team of five engineers cannot train everyone on everything in a single quarter. When a gap cannot be closed, record it explicitly:

• The service and restart path.
• The current evidence level.
• The accepted risk, in plain language.
• A review date.

The alternative is to leave the service silently unowned, which is the condition the audit was meant to find. An accepted risk with a review date is a decision. An unrecorded gap is an accident waiting for a trigger.

For teams running on AWS, GCP, or bare metal, the audit is the same exercise. The AWS Well-Architected Framework reliability pillar notes that achieving reliability can be challenging in traditional on-premises environments due to single points of failure, lack of automation, and lack of elasticity. The bus factor audit is a way to find the human single points of failure that architecture diagrams do not show.

A short, repeatable audit cadence

For a 2–15 engineer team, a quarterly cadence is enough to keep the map current without turning it into a project. The cadence can be as simple as:

1. Pick the three services with the highest blast radius.
2. For each, list the restart paths and the people who can execute them.
3. Mark the evidence level for each person.
4. Identify one gap per service and schedule a drill or shadowing session.
5. Update the map with the drill result and a new review date.

The audit is not a one-time report. It is a working artifact that changes as the team changes. The goal is not to eliminate every gap. The goal is to know where the gaps are, so that when the pager fires at 03:00, the person who answers has a realistic path to recovery.

Frequently asked questions

How is a bus factor audit different from a runbook review?

A runbook review checks whether the documentation is accurate. A bus factor audit checks whether a person can execute the documented steps. A runbook can pass review and still leave you with a bus factor of one if no one on the current team has executed it.

How many people need to be able to restart a service?

There is no universal number. The audit’s job is to make the current state visible, not to enforce a threshold. For a team of 2–15 engineers, the practical question is whether the service has at least one person who can execute the restart path when the primary is unavailable, and whether that person has evidence beyond reading the runbook.

What if the only person who can restart a service is leaving?

That is the highest-priority gap the audit can find. The response is a drill or shadowing session before the person leaves, with the goal of moving at least one other person to “has done it in a drill.” If that is not possible, the gap should be recorded as an accepted risk with a review date and a plan for the next quarter.

Should the audit include break-glass access?

Yes. Break-glass access is a restart path. If the normal credentials are unavailable, the break-glass path is the one that matters. The audit should record who can use it and whether they have tested it.

How often should the map be reviewed?

Quarterly is a reasonable default for a lean team. The map should also be reviewed after any incident that required a restart, after any on-call rotation change, and after any playbook update that changes a restart path.

For a related practice, see Write the Recovery Checklist Before You Need It, which covers the checklist side of the same problem: what to have ready before the restart is needed.

Most handover notes fail the same way. The author writes a careful description of the backup system — where the repository lives, which tool runs nightly, roughly how a restore works — and the reader, three days into a vacation, discovers the note describes intent rather than a path. The repository is reachable. The command is almost right. The backup set the reader picked cannot actually be restored to the target they need, and nothing in the note said so.

The unit of value in a handover note is not the description of the backup system. It is the executable recovery path: the exact command, the inputs it needs, and the observable that proves it worked. If the substitute cannot run it, the note has failed regardless of how complete it reads.

This is a craft note for lean teams — two to fifteen engineers, no dedicated SRE department, one on-call rotation — about what to write down before the person who can restore the database becomes unreachable for two weeks. It is not a template to fill in. It is a set of decisions about what belongs in the note and what does not.

Start from the failure mode, not the format

The failure mode is specific. The author leaves. A restore is needed. The substitute opens the note, finds a paragraph explaining that PostgreSQL maintains a write ahead log in pg_wal/ that records every change to the database’s data files, and finds a command that looks plausible. They run it. It completes. They start PostgreSQL. It comes up. The table they needed is still missing, because the backup they restored ended after the data was lost, and PostgreSQL can only play forward, not backward.

That is not a hypothetical. The pgBackRest user guide documents exactly this scenario in its point-in-time recovery walkthrough: if the wrong backup is selected, recovery to the required time target fails, and the log output does not make the failure obvious. The key is to look for the presence of the “recovery stopping before…” and “last completed transaction…” log messages. If they are not present, the recovery to the specified point-in-time was not successful.

A note that does not tell the reader to check for those log lines has not equipped them to know whether they succeeded. A note that does not tell them which backup sets are valid together has not equipped them to choose.

What the note must contain for each recovery scenario

For each scenario the team actually cares about — full cluster restore, point-in-time recovery to a timestamp, single-database restore, file-level restore from the object store — the note needs four things:

1. The exact command, with the flags that matter, not a paraphrase.
2. The inputs the command needs: which repository, which backup set, which target, which key material, which host.
3. The dependency chain: what else must be valid for this command to produce a usable result.
4. The observable that proves it worked — a log line, a query result, a checksum, a row count.

Anything else is context. Context is useful, but it is not the note. The note is the four items above, written so the reader’s first hour is spent executing rather than interpreting.

The PostgreSQL and PITR path

Continuous archiving combines a file-system-level backup with backup of the WAL files. Recovery restores the file system backup and then replays the backed-up WAL files to bring the system to a current state. That is the mechanism. The note’s job is to make the mechanism’s prerequisites explicit.

The first prerequisite is a continuous sequence of archived WAL files extending back at least as far as the start time of the backup. Not “some WAL files.” A continuous sequence, back to the backup’s start. If there is a gap — an archiving failure that was not noticed, a repository that filled, a version mismatch that stopped archiving — the backup is not recoverable to a point after the gap. The note should say where to check for gaps and what a gap looks like.

The second prerequisite is that pg_dump and pg_dumpall are not part of this path. They do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. If the team has both a logical dump and a physical backup, the note should say which one is for which purpose, because a substitute under pressure will reach for the one they recognize.

The third is that WAL archiving does not restore changes made to configuration files — postgresql.conf, pg_hba.conf, pg_ident.conf. Those are edited manually rather than through SQL operations. The note should say where the current configuration lives and how it is restored, because a cluster that comes up with the wrong pg_hba.conf is a cluster that comes up but does not accept the connections the application needs.

The fourth is the disk-space failure mode. If the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until space is freed. The note should say which filesystem to check and what the safe threshold is, because the substitute may be the person who has to free space at 2 a.m.

The pgBackRest path

pgBackRest’s restore command defaults to selecting the latest backup from the first repository where backups exist. That default is convenient and dangerous. The note should state it, because a reader who assumes the tool will pick the right backup will not pass --set or --repo, and the default may not be the backup they need.

For point-in-time recovery of --type=time or --type=lsn, pgBackRest selects a backup whose stop time is before the target time, or whose stop LSN is less than or equal to the target LSN. If no matching backup is found, it raises an error for both types; it does not fall back to the latest backup. That behavior belongs in the note verbatim, because the failure mode is a reader who assumes a fallback exists and does not check the error.

The dependency chain is the other thing the note must state. A differential backup requires both the differential backup and the previous full backup to be valid in order to restore. An incremental backup requires all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup to all be valid. The note should name which sets are valid together for the scenarios the team cares about, rather than pointing at “the backup.”

Two operational facts belong in the note as well. pgBackRest encrypts the repository based on a user-provided password, so the password is part of the recovery path, not a separate concern. And the local and remote pgBackRest versions must match exactly; if there is a mismatch, WAL archiving and backups will not function until the versions match. A note that does not record the version in use at the time of writing will be wrong after the next upgrade.

The restic and age path

For file-level backups, the note’s job is to separate two claims that are easy to conflate: “the repository exists” and “this person can decrypt and restore it.” Only the second one matters during a vacation.

age is a file encryption tool, format, and Go library with small explicit keys, post-quantum support, no config options, and UNIX-style composability. Files can be encrypted to multiple recipients by repeating the recipient flag, and every recipient will be able to decrypt the file. Identity files contain one or more secret keys, one per line, or an SSH key.

The note should record: where the repository lives, which recipients the backups are encrypted to, where the identity files are stored, and how the substitute obtains access to them. If the identity file is itself passphrase-protected and stored remotely, the note should say so, because that is a second access step that a substitute will not discover until they need it.

The note should also record what a successful restore looks like. For a restic repository, that is a restic check result and a test restore of a known file. For an age-encrypted archive, that is a decryption that produces the expected file. The observable is the point.

Break-glass access and offboarding hygiene

The handover note is where access hygiene and recovery access meet, and the tension is real. Least privilege says the substitute should not hold standing credentials they do not need. Recovery says the substitute must be able to reach the repository and the key material when the author is unreachable.

The resolution is to write down the break-glass path rather than to grant standing access. The note should name: the mechanism by which the substitute obtains elevated access, who approves it, how long it lasts, and how it is revoked. It should name the offboarding step that removes the author’s access when they return, so the break-glass grant does not become permanent. And it should name the person who can approve the grant when the author is the one who normally would.

This is the part of the note most likely to be skipped, because it is administrative rather than technical. It is also the part most likely to be needed, because a substitute who cannot reach the repository cannot run any of the commands the rest of the note describes.

The pre-vacation drill

The note’s only real test is running it as the substitute would. Not reading it. Running it.

The author picks a scratch target — a throwaway instance, a staging cluster, a container — and executes the note’s commands in the order the note presents them, using only the note as a guide. Every place the author has to supply knowledge that is not in the note is a gap. Every command that fails because a prerequisite was not stated is a gap. Every observable the author checks without the note saying to check it is a gap.

The record of that drill is the note’s revision history. It is more useful than a version number, because it says what was tested and when. A note that was drilled against pgBackRest 2.59.3 and PostgreSQL 17 is a different artifact from a note that was written against an earlier version and never re-run.

The drill also surfaces the failure behavior the note needs to record. A substitute who has run the commands once, on a scratch target, with the author watching, is a substitute who will recognize the difference between a restore that succeeded and a restore that completed.

Review, ownership, and re-verification

The note should have a named reviewer and a date. The reviewer is not a second author; they are the person who reads the note as a substitute would and reports what they could not follow. The date is the point from which re-verification is due.

The re-verification trigger is tied to change, not to a calendar. A note that was true at one pgBackRest version may be false at the next, because the local and remote versions must match exactly. A note that was true before a topology change may point at a repository that no longer exists. A note that was true before a key rotation may reference an identity file that has been replaced. The note should say what changes invalidate it, so the team knows when to re-drill.

This is the same discipline the SRE literature applies to postmortems. A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring. An unreviewed postmortem might as well never have existed. The same is true of a handover note. An unreviewed note is a document, not a recovery path.

It is worth defining, before the vacation rather than during it, what triggers a note review. The SRE book’s guidance on postmortem criteria applies here: define the criteria before the event so everyone knows when the process is necessary. For a handover note, the criteria are simpler — a version change, a topology change, a key rotation, or a drill that failed.

What the note is not

The note is not a postmortem. A postmortem explains what happened and why; a handover note explains what to do next. The two share a discipline of specificity and a refusal to blame, but they serve different readers at different times.

The note is not a runbook for every service. It is the recovery path for the things the author uniquely knows how to restore. If the team already has a runbook that a substitute can follow, the note should link to it rather than duplicate it. The Recovery Checklist Before You Need It is a useful companion for the general shape of a recovery checklist; the handover note is the specific, dated, drilled instance of it for one person’s absence.

The note is not a knowledge dump. A note that explains the history of the backup system, the reasons for each architectural choice, and the tradeoffs considered is a design document. Those are valuable. They are not what a substitute reads at 2 a.m. The note should be short enough to read in the first hour and specific enough to execute in the second.

Frequently asked questions

How long should the handover note be?

Long enough to contain the executable recovery paths for the scenarios the team cares about, and no longer. For a single PostgreSQL cluster with pgBackRest and a restic repository, that is typically a few pages: one section per scenario, each with the command, the inputs, the dependency chain, and the observable. Length is not the measure. The measure is whether a substitute can run it without asking the author a question.

Should the note include the backup password or key material?

No. The note should say where the key material is stored and how the substitute obtains access to it, not contain the material itself. pgBackRest encrypts the repository based on a user-provided password, and age identity files contain secret keys; both belong in a secrets manager or an equivalent access-controlled store, with the note recording the path and the approval step. A note that contains the key is a note that leaks when it is shared.

What if the substitute has never run a restore before?

Then the pre-vacation drill is not optional. The author runs the note’s commands as the substitute would, on a scratch target, and records what broke. The substitute should run them too, with the author available but not intervening. A note that has only been read is a hypothesis; a note that has been executed is a procedure.

How often should the note be re-verified?

Tie re-verification to change rather than to a calendar. A pgBackRest version change, a PostgreSQL major-version upgrade, a repository migration, a key rotation, or a topology change all invalidate parts of the note. The note should name those triggers so the team knows when to re-drill. A calendar-based cadence is a fallback for teams that cannot detect the changes, not the primary mechanism.

Does the note replace a postmortem when a restore fails?

No. If a restore fails during the author’s absence, that is an incident, and it merits a postmortem. The SRE book’s guidance is that postmortems are expected after any significant undesirable event, and that a blameless postmortem focuses on identifying the contributing causes without indicting any individual or team. The handover note is what the team uses during the incident; the postmortem is what the team uses afterward to make the next note better.

What is the single most important thing to write down?

The observable that proves the restore worked. Commands can be copied from documentation. The dependency chain can be reconstructed from the tool’s behavior. But the observable — the log line, the query result, the checksum — is the thing that tells the substitute whether they are done or whether they have produced a successful-looking but useless result. Write that down first, and write it for every scenario.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

Start with the rectangle. On the Composition Notebook Costume Teacher Group Halloween shirt, the printed artwork is a black-and-white speckled notebook-cover rectangle with a dark binding strip down the left side. Inside that rectangle sits a white ruled label headed COMPOSITION, and inside the label, an outlined ghost and an outlined pumpkin. The shirt is a printed notebook cover, not a physical notebook. That distinction matters more than it first appears.

We keep a small decision log at Gray Haven Lab, and one recurring entry is about surfaces. A runbook is a surface. A status page is a surface. A notebook cover is a surface too, but a strange one: it is the part of the object you see when the object is closed. The printed design on this shirt reproduces the closed state. It does not show pages, handwriting, or a spine cracked from use. It shows the cover as a graphic, which means the joke — if it is a joke — is about recognition rather than use.

That is the useful argument here. A composition notebook cover is one of the few pieces of school stationery that functions as a label before it functions as a tool. The speckled black-and-white pattern, the dark binding strip, and the ruled label are not decorative choices; they are the visual grammar of a mass-produced object that generations of students and teachers can identify from across a room. When that grammar is transferred to a shirt, the wearer becomes the closed object. The ghost and pumpkin sit inside the label where a name, subject, or date would normally go. The design does not ask you to read a slogan. It asks you to recognize a format.

I have written before about the difference between a slogan and a usable promise. A slogan is a phrase that sounds like a commitment but does not specify a trigger, an owner, or a failure mode. A usable promise is narrower and more honest: it says what will happen, under what conditions, and what it will cost. The composition notebook cover is closer to the second category, even though it contains no promise at all. It is a format with rules. You know what the label is for. You know where the binding is. You know the object is closed. The ghost and pumpkin are the only variables, and they are small enough to read as a seasonal annotation rather than a claim.

That is why the design works as a group costume without needing a caption. The shirt does not say “teacher” or “student” or “writer.” It says notebook. The viewer supplies the rest. In a group setting, the repetition of the same cover across several people turns the format into a pattern, and the pattern does the work that a printed phrase would otherwise have to do. This is not a historical claim about costume design; it is an observation about how the artwork reads. The notebook cover is recognizable enough that the ghost and pumpkin can be minor details rather than the main event.

There is a practical tradeoff in wearing a design this quiet. A shirt with a large slogan announces its intent immediately. A shirt with a notebook cover asks the viewer to complete the reference. Some viewers will not. That is the cost: the design is legible to anyone who has handled a composition notebook, and nearly invisible to anyone who has not. For a teacher group costume, that tradeoff is probably acceptable, because the group supplies the context. For an individual wearing it off-duty, the shirt reads as a graphic rather than a statement. Both readings are consistent with the artwork.

What I find interesting is the label. The white ruled rectangle headed COMPOSITION is the part of the design that carries the most information, and it is also the part that is easiest to overlook. The ghost and pumpkin are inside it, which means the seasonal elements are contained by the format rather than placed on top of it. That containment is what keeps the design from becoming a generic Halloween graphic. The notebook cover is not a background for a ghost; the ghost is an entry in the notebook. The distinction is small, but it changes how the shirt reads. One version is a costume. The other is a visual joke about where the costume is recorded.

We keep a decision log because memory is unreliable under pressure. A notebook cover is a reminder that the log has a front and a back, a binding and a label, and that the closed state is still part of the object. The shirt reproduces that closed state faithfully enough to be recognized. It does not claim to improve anything, teach anything, or solve anything. It is a printed surface that borrows the authority of a familiar format. That is a modest thing for a shirt to do, and it is enough.

What we’d do differently

If we were writing the product description, we would lead with the visual observation rather than the occasion. The notebook cover is the object; the teacher group costume is one use. Leading with the occasion makes the design sound like a prop, when the more interesting fact is that the artwork treats the cover as a complete graphic. We would also avoid implying that the shirt is a notebook or that it comes with one. The design is a printed image of a closed notebook cover, and the ghost and pumpkin are inside the label. That is the whole of it.

A break-glass account is a pre-staged emergency credential. Its value is not that it logs in fast. Its value is that when it logs in, everyone knows, and the session can be traced back to a named human afterward. For a team of two to fifteen engineers who run their own on-call rotation, that distinction matters more than it does in an organization with a dedicated security operations center.

The failure mode the account exists for is specific: the identity provider is down, the MFA device is lost, the last global administrator has left, or the federation path is unreachable. Microsoft’s guidance on emergency access accounts in Entra ID names these cases directly and states the rule plainly: use emergency access accounts only for emergency or break-glass scenarios where normal administrative accounts cannot be used, and restrict their use to times when it is absolutely necessary. The same logic applies whether your control plane is Entra ID, a self-hosted LDAP directory, or a cloud provider’s IAM.

Design constraints for a small team

Microsoft recommends creating two or more emergency access accounts, as cloud-only accounts on the *.onmicrosoft.com domain that are not federated or synchronized from on-premises. The redundancy is not decorative. A single emergency account is a single point of failure that happens to be the thing you reach for when everything else has already failed.

The federation independence is the load-bearing part. If your emergency account authenticates through the same identity provider that just went down, you have not built an emergency path. You have built a second door with the same broken lock. For a bare-metal or self-hosted environment, the equivalent is a local account that does not depend on the central directory, with credentials stored somewhere that does not require the network to reach.

Microsoft also specifies that emergency access accounts should be excluded from Conditional Access policies that block or restrict sign-in, because an enforced policy could prevent sign-in during the exact emergency the account is designed for. Report-only policies do not block access and do not need an exclusion. This is a tradeoff worth stating plainly: you are deliberately carving a hole in your normal access policy. The hole is the control. It is not a bug to be closed later.

The authentication method for the emergency account should differ from the methods used by normal administrative accounts. If your normal admin accounts use an authenticator app, the emergency account should use a FIDO2 security key or certificate-based authentication. The credential or device must not expire or fall under automated cleanup for lack of use. An emergency account that gets swept by a dormant-account policy is an emergency account that will not work when you need it.

Make the account conspicuous by design

Yale’s HIPAA break-glass procedure, which draws on a 2004 whitepaper from the Joint NEMA/COCIR/JIRA Security and Privacy Committee, describes the design principle well: emergency accounts should be created in advance so that careful thought can go into the access controls and audit trails associated with them, and the username should be obvious and meaningful, such as breakglass01, so that the account name would be inappropriate under normal operations and would stand out in audit trails.

That is the opposite of how most credentials are named. Normal accounts blend in. The emergency account should not. When someone greps the audit log six months from now, the account name should be the first thing that catches their eye.

The distribution method should also be physically evident. Yale’s procedure lists several options: kept behind glass in a cabinet where access requires literally breaking the glass, maintained within sealed envelopes where a broken seal is an obvious indication of access, locked in a desk drawer that only specific people can access, or locked in a safe or cabinet where one person knows the combination and a different person has the key to the room. The common thread is that access to the credential leaves a physical trace.

For a small team, the two-person pattern is practical. One engineer holds the safe combination; another holds the office key. Neither can reach the credential alone. The friction is the point. It is not a bug to be optimized away.

The distribution procedure should include a sign-out method requiring an acceptable form of identification, recorded before the accounts are made available. Yale’s procedure notes that this assures activities performed using the emergency account may eventually be associated with an authorized individual, creates accountability, and can assure non-repudiation. In practice, that means a paper log or a shared document with a timestamp, a name, and a reason. The reason field is not bureaucracy. It is the first line of the postmortem.

Wire the alert before you need it

Microsoft’s guidance is direct: monitor sign-in and audit log activity from the emergency accounts and trigger notifications to other administrators, so that you can verify these accounts are only used for testing or actual emergencies. The alert is the account’s real security control. If a break-glass sign-in does not page a human within minutes, the audit trail is archaeology rather than detection, and the account is effectively a permanent backdoor with a nice name.

For a team without a SOC, the alert should route to the same on-call channel that handles production incidents. The alert should fire on every sign-in, not on a threshold. There is no legitimate reason for a break-glass account to sign in without a human knowing about it in real time.

The audit record itself should contain enough to reconstruct the event. NIST SP 800-53 Rev. 5, as reflected in the CSF reference, specifies that audit records should establish what type of event occurred, when it occurred, where it occurred, the source of the event, the outcome, and the identity of any individuals, subjects, or objects associated with the event. For a break-glass session, that means the account name, the source IP or workstation, the timestamp in UTC, the actions taken, and the outcome. If your system cannot produce that, Yale’s procedure is explicit: break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead.

NIST SP 800-92 frames log management as the process for generating, transmitting, storing, analyzing, and disposing of computer security log data, and notes that it is essential to ensuring that computer security records are stored in sufficient detail for an appropriate period of time. For a small team, the practical version is: ship the break-glass account’s logs to a destination that the account itself cannot modify, and keep them long enough to cover your incident review cycle.

Write the post-use runbook

After any use of an emergency access account, Microsoft’s guidance calls for a review to determine whether the use was authorized and whether the actions taken were appropriate. That review has a few concrete steps.

First, preserve the logs. Capture the sign-in record, the audit trail of privileged actions, and any session recordings or command history the platform provides. NIST’s audit guidance specifies that audit record reduction and report generation should support on-demand review and after-the-fact investigations without altering the original content or time ordering of audit records. Copy the raw records before you summarize them.

Second, reconcile the session to a named human. The sign-out log from the distribution procedure gives you the name. The audit trail gives you the actions. The postmortem gives you the reason. If those three do not line up, that is the finding.

Third, disable or rotate the credential. Yale’s cleanup procedure recommends disabling or deleting the emergency account that was used to prevent reuse now that the password is known, and reconciling the data acquired and audit trails to reflect the proper operator’s name. Some systems can automatically deactivate emergency accounts after first use or after a selectable period such as 8 hours or 1 day. Commvault’s break-glass implementation takes a different approach: the password is regenerated on every login to prevent repeated use of the same password, and all actions are recorded in the audit trail. Either pattern works. What does not work is leaving a known password in place after an incident that involved a dozen people in a channel.

Fourth, ask whether the normal access path failed in a way that should be fixed. Yale’s procedure notes that frequent use of emergency accounts may indicate problems with the normal user authentication mechanism. A break-glass account that gets used every month is not a break-glass account. It is a helpdesk with extra steps, and Yale’s procedure explicitly says break-glass should not replace a helpdesk.

Schedule the drill

Microsoft recommends performing account validation drills at least every 90 days, including reviewing the authorized-user list and testing sign-in and administrative task functionality. The 90-day cadence is a reasonable default for a small team. The important part is the second half of that sentence: testing administrative task functionality, not just sign-in.

A drill that only tests sign-in is not a drill. The failure that matters is discovering at 3 a.m. that the credential works but the permission it grants does not cover the task. If the break-glass account is supposed to let you restore a database, the drill should include a restore. If it is supposed to let you rotate an IAM role, the drill should include a role switch. AWS IAM documentation notes that when you switch to a role in the console, you temporarily give up your user permissions and work with the permissions assigned to the role, and when you exit the role, your user permissions are automatically restored. That is a useful property for a break-glass path, but only if the role’s permissions actually cover the emergency task.

One more AWS-specific detail worth knowing: you cannot switch roles in the AWS Management Console to a role that requires an ExternalId value. Such a role can only be assumed by calling the AssumeRole API that supports the ExternalId parameter. If your break-glass path depends on a role with an ExternalId, the console switch will not work, and the drill will find that out before the incident does.

Treat a failed drill as an incident. It is a control that did not work when tested under controlled conditions. The blameless postmortem format applies: what did we expect, what happened, what will we change.

The tradeoff to accept

The break-glass account is slower and more awkward than a normal login on purpose. Two people, a physical envelope, a sign-out log, a page to the on-call channel, a postmortem the next morning. Every one of those steps is friction, and every one of those steps is the control.

The alternative is an account that is fast and quiet. That account is not a break-glass account. It is a standing privilege with a reassuring name, and the first time it is used without anyone noticing, you will have lost the ability to tell an emergency from a misuse.

If you are building the recovery checklist that the break-glass account supports, the Recovery Checklist Before You Need It is a useful companion. The break-glass account gets you into the system. The checklist tells you what to do once you are there.

Questions that come up

How many break-glass accounts should a small team have? Microsoft recommends two or more for redundancy. For a team of two to fifteen, two is the practical minimum. One is a single point of failure. More than two multiplies the credential storage and rotation burden without adding much resilience.

Should the break-glass account be a person or a shared account? Microsoft’s guidance says not to associate emergency access accounts with any individual user in the organization, and to store credentials in a known secure location available to multiple members of the administration team. The account is a shared emergency path, not a personal login. The sign-out log is what ties a specific session to a specific person.

What if the system cannot produce an audit trail? Yale’s procedure is explicit: if the system cannot provide an audit trail showing simple account activity like login attempts, break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead. A break-glass account without an audit trail is a standing privilege with no accountability.

How often should the credential be rotated? The sources describe per-login regeneration (Commvault) and post-use disable-or-delete (Yale), but do not specify a fixed rotation interval for offline-stored credentials. The 90-day validation drill is the cadence the sources support. Rotation after every use is the safer default; a fixed calendar rotation is a secondary control.

Does the break-glass account replace normal access controls? No. The sources frame it as a last resort alongside standard controls. Yale’s procedure says standard access controls should be established with sufficient rules to minimize the number of times break-glass needs to occur. The account is the exception path, not the primary one.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The garment preview shows one word: meh. Lowercase. Bold. White on a dark shirt. No animal. No subtext. No parenthetical. The period is the only punctuation, and it lands like a closed door.

I keep looking at it because it refuses to do the thing most slogan shirts do. It doesn’t ask a question. It doesn’t list a hobby. It doesn’t wink at a fandom. It states a verdict and stops. For a team that spends its days writing runbooks, escalation policies, and decision records, that restraint is worth studying.

The shirt is listed as Meh Funny Sarcastic Saying Introvert Dry Humor under the Quirky Cotton Collective brand. The retail title stacks three descriptors — funny, sarcastic, introvert, dry humor — but the artwork itself carries none of that scaffolding. The design is the word. Everything else is metadata.

What the artwork actually does

Large bold lowercase typography on a dark ground is a visual choice with consequences. Lowercase reads as casual, unbothered, almost mumbled. Bold weight makes it loud. White on dark gives it contrast. The combination is a contradiction: a shout that says nothing is worth shouting about.

That contradiction is the joke, if there is one. The word meh is a shrug. Rendering it in the largest available type is a way of shrugging theatrically. The shirt doesn’t claim the wearer is indifferent to everything; it claims the wearer has decided this particular moment — the moment of reading the shirt — does not merit enthusiasm.

I’m not going to pretend this is a deep statement about labor, burnout, or the human condition. It’s a word on a shirt. But the word is doing something specific: it is refusing to promise anything. And that refusal is the most interesting thing about it.

Slogans versus promises

In operations work, we spend a lot of time distinguishing between a slogan and a usable promise. A slogan is a phrase that sounds good in a meeting. A usable promise is a sentence you can test. “We prioritize reliability” is a slogan. “We page a human within five minutes of a failed synthetic check” is a usable promise. One of them survives an incident; the other becomes a postmortem bullet point.

The meh shirt is not a usable promise. It is not trying to be. It is a slogan that knows it’s a slogan. It doesn’t claim the wearer is calm under pressure, or that they have a five-nines uptime record, or that they never lose their temper in a code review. It claims they are unimpressed. That’s a mood, not a metric.

I find that honesty refreshing. Most slogan shirts overpromise. They imply a personality trait — “I’m the funny one,” “I’m the smart one,” “I’m the one who fixes your computer” — and then the wearer has to live up to it. The meh shirt implies nothing. It is a preemptive withdrawal from the conversation. You can’t be disappointed by someone who never claimed to be impressive.

The introvert framing

The retail title includes “Introvert,” and the related search queries include “meh introvert shirt.” I understand why. The word meh functions as a boundary marker. It says: I am here, I am participating at the minimum required level, and I would like to go home now. For people who find social interaction draining, that is a recognizable posture.

But the artwork itself does not say “introvert.” It says meh. The introvert reading is an interpretation, not a printed claim. I think that’s important. The shirt is not a diagnosis. It is a mood that happens to align with a certain temperament. You can wear it as an introvert, an extrovert, or someone who just doesn’t want to talk about the weather. The word doesn’t check your credentials.

In team settings, this matters. A shirt that says “I’m an introvert” invites a conversation about personality types. A shirt that says meh invites nothing. It closes the loop. For someone who wants to signal a boundary without explaining it, that’s efficient.

What it doesn’t do

I want to be clear about what this shirt is not. It is not a diagnostic tool. It does not measure burnout, introversion, or operational fatigue. It does not cure anything. It does not improve team communication. It is a piece of clothing with a word on it.

I also can’t tell you how it fits, how it wears, or how it holds up after a wash. I haven’t worn it. I’m looking at the artwork as observed in the native garment preview: large bold white lowercase meh. lettering on a dark shirt, no animal illustration present. That’s the object. Everything else is inference.

What I can say is that the design is legible. The word is the whole design. There is no competing element, no decorative flourish, no hidden meaning. It is a single word, rendered large, and it does not pretend to be more than that.

Why this matters for technical readers

If you work on a small team, you are probably familiar with the gap between what a system promises and what it delivers. You write a runbook that says “restore from backup,” and then you discover the backup is corrupt. You write an escalation policy that says “page the on-call,” and then you discover the on-call is asleep. The gap between slogan and promise is where incidents live.

The meh shirt is a reminder that not everything needs to be a promise. Some things are just statements. Some things are just moods. Some things are just a word on a shirt. And that’s fine. Not every object has to carry the weight of a reliability practice.

But I do think there’s a small lesson here about honesty in language. The shirt doesn’t overclaim. It doesn’t say “I’m fine.” It doesn’t say “I’m thriving.” It says meh. That’s a low bar, and it clears it. In a world of overpromising slogans, there’s something to be said for a garment that promises nothing and delivers exactly that.

What we’d do differently

If we were designing a shirt to communicate a boundary, we would probably not choose a single word. We would choose a sentence with a subject and a verb, something testable. But that would be a different object. The meh shirt is not a boundary statement; it is a mood statement. And moods don’t need to be testable.

What we would do differently is resist the urge to over-interpret it. It is a word on a shirt. It is not a manifesto. It is not a diagnosis. It is not a promise. It is a shrug, rendered in bold lowercase type, and it is content to be exactly that.

Sometimes the most honest thing you can print is the thing you don’t bother to capitalize.

A runbook can be correct when written, still linked from the alert that invokes it, and wrong in the one step that matters. For a team of two to fifteen engineers who are their own on-call rotation, that is the expensive version of staleness: the document looks authoritative, the alert keeps firing, and the person reading it at 3 a.m. has no way to know which paragraph has quietly stopped matching production.

This is not a documentation problem in the abstract. It is a detection problem. Staleness is a property of the relationship between a runbook and the system it describes, not of the runbook’s age. A six-month-old runbook for a service that has not changed may be perfectly fresh. A two-week-old runbook for a service that shipped a new deploy path may already be wrong. The goal is to make that decay surface as a routine chore on a Tuesday afternoon rather than as a surprise during a customer-visible outage.

Why staleness is invisible by default

Runbooks are read under pressure, not on a schedule. The alert that points to one keeps firing, which creates a false sense of validation: the runbook is still reachable, so it must still be right. Nothing in the normal workflow compares the document to the system it describes. The person who wrote it may be the only person who has ever executed it, and their memory silently fills the gaps a fresh reader would hit immediately.

Google’s SRE Workbook is direct about the underlying rate of change. In its chapter on on-call, it states that playbook details go out of date at the same rate as the production environment changes, and that for daily releases a playbook might need an update on any given day. The same chapter notes that in SRE, whenever an alert is created, a corresponding playbook entry is usually created, and that these guides reduce stress, mean time to repair, and the risk of human error. That is the case for having runbooks. It is also the case for not trusting them indefinitely.

The workbook offers two broad strategies for keeping playbooks maintainable. One is to keep entries general enough that they change slowly — for example, a single entry for a class of alerts that a trained on-call engineer reads alongside an architecture diagram. The other is to write step-by-step procedures that reduce human variability and drive down MTTR. Both are legitimate. The workbook’s practical advice is that if a team agrees on nothing else, it should decide what minimal, structured details its playbooks must have, and notice when playbooks have accumulated a lot of information beyond those structured details. That minimal structured set is also the natural place to hang freshness signals.

Borrow the postmortem discipline

Google’s postmortem chapter contains a line that applies almost verbatim to runbooks: an unreviewed postmortem might as well never have existed. The chapter encourages regular review sessions so that each completed draft is actually reviewed, and it stresses that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary. It also lists a monitoring failure — which usually implies manual incident discovery — among the common triggers.

The same logic transfers. Define the runbook review trigger before you need the runbook. If the only time a runbook gets read is during an incident, then the incident is your review process, and you are paying for it in customer impact. A scheduled freshness check is cheaper.

Staleness signals worth checking

The following checklist is our own operational reasoning, not a measured result. It is offered as a template. The signals are chosen because each one can be checked mechanically or by a reader who did not write the runbook.

• Last-verified date and named owner. A runbook without an owner has no one to notice when it drifts. A date without a name is a timestamp, not accountability. A vendor guide on incident response runbooks recommends scheduling quarterly reviews, making version control standard, and keeping ownership visible so the runbook reflects the current state of the infrastructure. The cadence is a recommendation, not a finding; the ownership point is the durable part.
• The alert or dashboard the runbook is bound to still exists and still fires. If the alert was renamed, retired, or split, the runbook is orphaned even though its text is unchanged. Orphan detection is a mechanical check and can be automated with a link and asset linter.
• Every command still parses and its expected output still matches. A command that returns a different shape of output than the runbook says to expect is a stale runbook, even if the command itself succeeds.
• Every linked dashboard, log query, and escalation path still resolves. Dead links are the cheapest staleness signal to detect and the easiest to ignore.
• The rollback step still corresponds to the current deploy mechanism. Deploy paths change more often than most other parts of a service, and the rollback step is the one you least want to discover is wrong.
• The stated blast radius still matches the service’s current dependencies. A runbook that says “affects the checkout service” when the service now also fronts authentication is understating the incident.

One more signal is worth treating as a smell in its own right: if the runbook is a fixed list of commands run every time a particular alert fires, that is a sign the runbook is standing in for automation. The SRE Workbook makes this explicit — if playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, it recommends implementing automation. A runbook that should be a script will drift every time the underlying command surface changes, and it will drift silently.

A cheap freshness drill for a small team

Pick one runbook per week. Have someone who did not write it execute it against a non-production or read-only path. Record what broke. That is the whole drill.

This is a rehearsal of the document, not of the recovery. Keep it separate from your backup and recovery drills so neither dilutes the other. A restore lottery tests whether your backups can be restored; a runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways and should not share a slot.

The cadence should be sized to the team, not copied from a large-org quarterly cycle. One runbook per week per rotation is a workable default for a small team. The point is that the cadence exists and is owned, not that it matches a specific number. If your team is already running a recovery checklist, the same weekly slot can host both, as long as the two exercises stay distinct. The Recovery Checklist article is a useful companion for the restore side; the freshness drill here is the documentation side.

Keep the freshness check read-only or non-production wherever possible. The goal is to detect drift, not to add a second, riskier change path to production.

What to do when a runbook fails its check

Treat it as an owned follow-up with a due date, not a documentation chore. The postmortem discipline of assigning follow-up actions applies equally to runbook maintenance. The difference is that runbook decay is predictable and can be scheduled rather than discovered. A vendor guide on incident response runbooks puts it plainly: treat every incident as a freshness test, and let stale instructions create an owned follow-up. That is the right instinct, but it is reactive. The scheduled drill is what makes it proactive.

If the runbook repeatedly fails its freshness check, that is information. Three options are available: rewrite it so the steps are stable, retire it and replace it with a pointer to the automation that now handles the case, or automate the deterministic parts and keep the runbook for the judgment calls. The SRE Workbook’s recommendation to automate deterministic command lists is the strongest signal that a runbook has outlived its form.

There is a related trap worth naming. PostgreSQL’s documentation notes that WAL archiving restores changes to the database but not changes to configuration files such as postgresql.conf, pg_hba.conf, and pg_ident.conf, since those are edited manually rather than through SQL operations. A runbook that says “restore from WAL” without saying where the configuration files come from is stale in a way that will only show up during a real recovery. The same documentation advises monitoring the archiving process to ensure it is working as intended, and warns that if archiving falls significantly behind, the amount of data that would be lost in a disaster increases. That is a monitoring signal, but it is also a runbook signal: if the runbook does not tell the responder how to check archive lag, the runbook is incomplete.

Where the published guidance already agrees

The SRE Workbook’s playbook section is the closest primary source for this argument. It acknowledges that playbook detail decays at the rate of production change, that teams should agree on minimal structured fields, and that deterministic command lists should be automated. The postmortem chapter supplies the review discipline: unreviewed artifacts might as well not exist, and criteria should be defined before the incident. A vendor guide on incident response runbooks adds the operational habits — scheduled reviews, version control, visible ownership, and treating incidents as freshness tests — though its comparative tool claims are marketing material and should not be read as independent evidence.

What none of these sources provides is a measured rate of runbook staleness in the field, or a measured effect of scheduled freshness checks on incident outcomes for teams of this size. That evidence does not appear to exist in the retrieved material. The checklist above is therefore a proposed practice, not a validated one. It is cheap enough to try, and the failure mode it addresses — a runbook that is wrong in the one step that matters — is expensive enough to justify the experiment.

FAQ

How often should a small team review runbooks?

There is no evidence-based cadence for a 2–15 engineer team. One runbook per week per rotation is a workable default because it distributes the load and keeps the review short. The important property is that the cadence is owned and visible, not that it matches a particular interval. A quarterly cycle copied from a larger organization will likely be too infrequent for a service that deploys daily.

Should the freshness check be part of the on-call shift?

It can be, but it should not be the on-call engineer’s only documentation task. The value of having someone who did not write the runbook execute it is that they hit the gaps the author’s memory fills. If the same person writes and checks every runbook, the check degrades into a reread.

What is the difference between a runbook freshness check and a restore drill?

A restore drill tests whether your backups can be restored. A runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways. A restore drill can pass while the runbook that tells you how to run it is wrong, and a runbook can be accurate while the backup it references is unusable. Keep them separate.

When should a runbook be retired rather than updated?

When the procedure it describes has been automated, or when the service it describes no longer exists. The SRE Workbook’s guidance to automate deterministic command lists is a useful trigger: if the runbook is a fixed list of commands run every time an alert fires, the runbook is a candidate for replacement by a script. Retiring it is not a loss; it is the intended outcome.

Does a stale runbook actually cause longer incidents?

The retrieved sources do not measure this. The SRE Workbook states that playbooks reduce stress, MTTR, and the risk of human error, which implies that a wrong playbook does not. But the specific claim that stale runbooks lengthen time-to-mitigation is not established by the evidence reviewed here. Treat it as a reasonable hypothesis, not a finding.

The honest limit

None of this prevents the incident where the runbook was fine and the system was not. The goal is narrower: make sure that when a runbook is wrong, you find out on a Tuesday afternoon and not during a customer-visible outage. For a lean team with no dedicated reliability coverage, that is a worthwhile trade.

Day-five deployability is a sequencing problem, not a permissions problem. The goal is not to hand a new engineer the keys to production; it is to stage access so that each grant is justified by a rehearsed task, and to make the first production deploy a supervised drill with a rollback path the engineer has already walked. For a team of two to fifteen engineers who are their own on-call rotation, the scarce resource is attention. Staging access so the new engineer can self-serve read-only investigation early buys back senior time that would otherwise be spent answering “what does this dashboard mean” questions.

Start with read-only observability, not deploy rights

The first grants should let the new engineer see the system without changing it. That means read-only access to logs, metrics, dashboards, and the deployment history. In AWS, this is a role assumed through federation with an identity provider, not an IAM user with long-lived credentials. AWS’s IAM best-practices page states plainly: “Require human users to use federation with an identity provider to access AWS using temporary credentials.” The same posture applies on GCP, where Google Cloud advises avoiding user-managed service account keys and using other authentication methods whenever possible.

Read-only access is not a courtesy. It is the prerequisite for every later grant. A new engineer who can query the last deploy’s commit SHA, the error rate during that window, and the relevant log lines can form a hypothesis before asking for production write access. That is the difference between a grant that is earned by a rehearsed action and one that is assigned by a checklist.

Stage the sequence by task rehearsal

The sequence that works for small teams looks like this:

1. Read-only observability and logs. The engineer can inspect production state but not mutate it.
2. Non-production deploy. The engineer deploys to staging or a scratch environment, using the same pipeline that production uses. This rehearses the deploy path without production risk.
3. Production deploy behind a supervised change. The first production deploy is paired with a rollback rehearsal and a synthetic check. The engineer sees the failure path, not just the happy path.
4. Break-glass and credential-issuing paths last. These are the grants that can cause the most damage if misused, and they are the least necessary for a first deploy.

Each step should be preceded by the new engineer demonstrating the read-only or non-production version of the task. A day-three task might be: restore a non-production database from the latest backup into a scratch environment and diff row counts against the source. That rehearses the restore path, surfaces whether the backup is actually restorable, and justifies the restore-credential grant with an observed action rather than a role name.

Default to temporary credentials and least privilege

AWS recommends granting only the permissions required to perform a task, starting broad only while exploring and then reducing toward least privilege. In practice, that means starting with a narrow role and widening only against observed activity. IAM Access Analyzer can generate least-privilege policies based on access activity logged in CloudTrail, which is a more reliable signal than a role template written from memory.

For human access, temporary credentials via federation are the default. Long-lived access keys are the exception, and the exception should be documented. AWS notes that some use cases require long-term credentials, such as workloads that cannot use IAM roles or third-party clients that do not support IAM Identity Center. Those are workload cases, not human onboarding cases. If a new engineer needs a long-lived key to deploy, that is a signal that the deploy path itself needs work.

On GCP, the equivalent discipline is avoiding user-managed service account keys. Google Cloud warns that service account keys can become a security risk if not managed carefully, and that the main threats include credential leakage, privilege escalation, and non-repudiation. The recommended mitigation is to avoid user-managed keys and use other authentication methods whenever possible. Where keys are unavoidable, Google Cloud recommends routinely rotating all keys you manage, including service account keys.

One tradeoff worth naming: setting an expiry time on a service account key can cause workloads to fail when the key expires. Google Cloud advises against expiry times for production workloads for this reason. The safer pattern for production is a key that does not expire, managed through rotation, rather than a key that expires and takes a service down with it.

Make the first production deploy a drill

The value of day-five deployability is not that the engineer has “full access.” It is that the team has a working rollback path and a synthetic check, and that the new engineer has walked both. A supervised first deploy should include:

• A pre-agreed rollback command or pipeline action, rehearsed in non-production.
• A synthetic check that fires if the deploy breaks a known path.
• A named person who is watching the deploy and can call the rollback.
• A short written record of what was deployed, what was checked, and what would have triggered a rollback.

This is where backup and recovery drills intersect with access sequencing. PostgreSQL’s documentation states that you should set up and test your procedure for archiving WAL files before you take your first base backup. The same logic applies to access: test the rollback path before you grant the deploy right. If the new engineer’s first production deploy is also the first time anyone has run the rollback in six months, the drill is doing double duty, and that is a feature, not a bug.

For file-level restores, restic’s restore command supports a --dry-run option that prints information about what would be restored without running the full restore. That is a useful rehearsal step for a new engineer: run the dry-run, read the output, then run the real restore into a scratch target. restic also warns that restoring data in-place can leave files in a partially restored state if the restore operation is interrupted, and recommends creating a current backup before restoring a different snapshot. A new engineer who has read that warning and practiced the safe path is better prepared than one who has only been granted the role.

Write the offboarding and review dates into the same ticket

If the grant has no scheduled review, it is a permanent grant in practice regardless of intent. AWS recommends regularly reviewing and removing unused users, roles, permissions, policies, and credentials. The practical way to do that for a small team is to put the review date and the offboarding path in the same ticket that grants day-one access. When the engineer leaves, or when the project ends, the ticket already contains the list of what to revoke.

This is also where blameless postmortem culture connects to access hygiene. Google’s SRE book describes blameless postmortems as a tenet of SRE culture that focuses on contributing causes without indicting an individual or team. The same posture applies to access reviews: the question is not “who granted this,” but “what task justified this grant, and is that task still happening?” Google’s SRE book also recommends defining postmortem criteria before an incident occurs so everyone knows when a postmortem is necessary. Defining access-review criteria before onboarding occurs serves the same purpose.

What can wait until after day five

Not every grant needs to be in place for a first deploy. The following can typically be deferred without blocking a supervised production deploy:

• Break-glass access to credential-issuing systems.
• Write access to production data stores, beyond what the deploy itself requires.
• Access to billing, cost-management, or account-level settings.
• Permission to modify IAM policies, roles, or service accounts.
• Access to backup repositories in a write or delete capacity.

Deferring these grants is not a lack of trust. It is a recognition that each one carries a different blast radius, and that the first week is better spent rehearsing the deploy and rollback paths than accumulating permissions that will not be exercised until later.

Monitoring and escalation access

A new engineer who is joining the on-call rotation needs to understand the escalation path before they need to use it. PagerDuty documents that integrations can be configured via email, API, or MCP, and that an email integration is created with its own email address. For a small team, the practical step is to add the new engineer to the escalation policy in a shadow capacity first: they receive the page, but a senior engineer is primary. That gives them the context of what pages look like without the pressure of being the first responder.

The same logic applies to synthetic checks. A new engineer should be able to see the check, read its history, and understand what a failure looks like before they are responsible for responding to one. If the team has a runbook for the check, the new engineer should read it and, ideally, update it with anything that was unclear. That is a low-risk way to contribute to operational knowledge while still building context.

FAQ

Does this mean a new engineer should not have production access in week one?

No. It means the production access should be scoped to the deploy path and paired with a rollback rehearsal. Read-only production access can and should come early. Write access to production should be tied to a specific, rehearsed task.

What if the team does not use AWS or GCP?

The provider-specific guidance cited here does not apply unchanged to bare-metal environments. The underlying principles, temporary credentials where possible, least privilege, scheduled review, and rehearsed rollback, are portable. The implementation will differ.

How does this relate to backup and recovery drills?

Access sequencing and recovery drills share the same logic: test the procedure before you rely on it. PostgreSQL’s documentation advises setting up and testing WAL archiving before taking the first base backup. restic’s --dry-run option lets you see what a restore would do before it does it. A new engineer’s first production deploy should follow the same pattern. For a broader checklist, see Write the Recovery Checklist Before You Need It.

What about break-glass access?

The retrieved sources do not describe break-glass procedures in detail. The general principle is that break-glass access should be the last grant, should be logged, and should have a defined review path. If your team has a break-glass procedure, the new engineer should read it before they need it, but they do not need to hold the credential on day one.

How do we know when to review access?

AWS recommends regularly reviewing and removing unused users, roles, permissions, policies, and credentials. For a small team, the practical trigger is the same ticket that granted the access. If the ticket has a review date, the review happens. If it does not, the grant tends to persist.

The sequence, in short

Read-only observability first. Non-production deploy second. Production deploy behind a supervised change with a rehearsed rollback third. Break-glass and credential-issuing paths last. Default to temporary credentials and least privilege. Write the review and offboarding dates into the same ticket that grants access. Treat the first production deploy as a drill, not a graduation. The point is not to slow the new engineer down; it is to make sure that when they deploy, they have already seen what happens when it goes wrong.

Editorial note: This publication and Quirky Cotton Collective share an owner. This is not an independent review.

We noticed the diagram before we read the words. On the I Had Potential shirt, a dashed empty circle sits at the upper left ledge. A solid white ball rests on the lower right flat ledge. Between them, a curved ramp descends. The phrase I HAD POTENTIAL runs across the top in white capitals.

That is the whole design. No arrows, no equations, no labels. The joke depends on the reader completing a physics problem in their head: a ball at the top of a ramp has gravitational potential energy; a ball at the bottom has converted it to kinetic energy and heat. The dashed circle marks where the ball used to be. The solid ball marks where it ended up. The phrase turns the diagram into a first-person statement about a person, not a particle.

We keep a small decision log for operational writing, and one recurring entry is the difference between a slogan and a usable promise. A slogan is a sentence that sounds like a commitment without specifying a trigger, an owner, or a measurable outcome. A usable promise names what will happen, by when, and who will notice if it does not. The shirt is a slogan. That is not a criticism. It is a diagram of the gap between the two.

What the drawing actually shows

The visual observation is specific: a curved descending ramp, not a straight diagonal. A dashed empty circle at the upper left ledge. A solid white ball at the lower right flat ledge. The curve matters because it changes the implied motion. A straight ramp suggests a simple slide, a constant acceleration down an inclined plane. A curved ramp suggests something closer to a brachistochrone or a roller-coaster profile, where the path bends and the ball’s speed changes non-uniformly. We are not claiming the designer ran the calculus. We are saying the curve reads as a deliberate shape, not a default diagonal line.

The dashed circle is the more interesting choice. A dashed outline conventionally indicates something absent, hypothetical, or no longer present. Here it marks the ball’s starting position. The solid ball marks the current position. The design does not show the ball mid-roll. It shows before and after, with the ramp as the implied interval. That is a compact way to draw a process: initial state, transition, final state. It is also, unintentionally or not, a diagram of a postmortem timeline.

The double meaning of potential

In physics, potential energy is stored energy associated with position. An object at height h in a gravitational field has potential energy mgh. At the bottom of the ramp, that energy has been converted. The word potential in the phrase I HAD POTENTIAL uses the ordinary English sense: capacity for future achievement. The diagram literalizes the physics sense. The pun works because both meanings describe a quantity that was present at the start and is absent at the end.

We would not claim this is a rigorous physics joke. A ball at the bottom of a ramp still has potential energy relative to the floor, or to the center of the Earth. The dashed circle could be read as a ball that never existed, or one that has already rolled away. The design leaves that ambiguous. The ambiguity is part of why it reads as a joke rather than a lecture.

What we find useful about the shirt as an object is that it does not pretend the transition was productive. The ball did not gain anything by rolling down. It lost height and gained speed, and the phrase frames that as a loss of potential. That is a pessimistic reading of a physics diagram, and it is also a familiar one to anyone who has watched a promising system degrade after a series of small, reasonable changes.

Why this reads as a boundary statement

Technical teams talk about potential constantly. Hiring conversations, promotion packets, architecture reviews, and incident postmortems all traffic in the language of what a system or a person could become. The shirt is a terse counterpoint. It says: the potential was there, and now it is not. It does not say why. It does not assign blame. It does not propose a fix. It just draws the before and after.

We keep a decision log entry about terse messages. A terse message can be a boundary statement or a passive-aggressive one, and the difference is usually whether it names a next action. I HAD POTENTIAL names no next action. It is a closed statement about a past state. On a shirt, that is fine. In a runbook, it would be a defect. The same sentence can be a joke in one context and an incident summary in another, and the diagram is what tips it toward joke.

The shirt does not claim to improve anything. It does not claim to cure, fix, or optimize. It is a printed observation. We are describing the artwork, not recommending it as a management technique.

What we would do differently

If we were writing a diagram like this for an internal runbook, we would label the axes. We would mark the initial height, the final height, and the energy lost to friction. We would add a timestamp. The shirt does none of that, and it is better for it. The missing labels are what let the reader supply the context: a project, a quarter, a career, a service that used to be simple.

What we would do differently in our own documentation is steal the before-and-after structure. A dashed outline for the previous state, a solid mark for the current state, and a single curved line between them. It is a compact way to show a transition without pretending the transition was neutral. We would add the labels the shirt omits. The shirt can keep the joke.