Gray Haven Lab

A third-party API starts returning 429s. Your queue backs up, your retries pile on, and by the time someone notices, the on-call engineer is staring at a dashboard that says “rate limited” and a vendor status page that says “all systems operational.” The incident ends. Now you have to write it up.

The temptation is to file a short postmortem that says “vendor throttled us, we backed off, resolved.” That document will not help the next on-call engineer, and it will not prevent the next incident. The useful postmortem for a rate-limit incident records three things the team can actually verify: what the dependency did, what your own client did, and what you changed afterward. Everything else is narrative.

This is written for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated reliability staff. The structure below assumes you already write postmortems; if you are still deciding what belongs in one, Google’s Postmortem Culture chapter defines the baseline: a postmortem is “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” A rate-limit incident fits that definition, but each section needs different evidence than a code-defect incident does.

Start with impact and timeline, not with the vendor

Record what users experienced before you record why. Which internal services degraded, for how long, and what the user-visible symptom was. A batch pipeline that fell two hours behind is a different incident from a checkout flow that returned errors for eleven minutes, even if both trace to the same 429.

The timeline needs two specific timestamps that teams often skip: when the first rate-limit response appeared, and when a human first knew about it. The gap between those two is your detection latency, and it is usually the most actionable number in the document. If the first 429 arrived at 02:14 and the first page fired at 02:51, the postmortem should say so plainly, because that thirty-seven-minute gap is a monitoring problem you can fix independently of the vendor.

Also record how the incident was detected. An alert on error rate, a synthetic check, a customer report, and a colleague noticing a stale dashboard are four different detection paths with four different remediation stories. Name the one that happened.

Log the dependency’s observed behavior verbatim

This is the section that saves the next on-call engineer the most time, and it is the one most often written from memory. Do not write it from memory. Paste the actual response.

For each rate-limit response you captured, record:

• The HTTP status code and the full response body or error payload, verbatim.
• Any rate-limit or retry headers the vendor returned, with their values.
• The request ID or trace ID from the response, if the vendor provides one, so a support ticket can reference it.
• The timestamp of the response, in UTC, and the endpoint or method that triggered it.

The payload matters because “429” is not a complete description. Google’s API design guidance, AIP-193, shows a 429 response carrying a RESOURCE_EXHAUSTED status and a machine-readable ErrorInfo block with a reason and domain. A vendor that follows that pattern gives your client something to branch on. A vendor that returns a bare 429 with an HTML error page gives you nothing, and that difference belongs in the postmortem because it shapes what your retry logic can safely do.

Then record what the vendor’s public documentation says about throttling, and label it as documentation rather than observation. AWS’s API Gateway documentation, for example, states that throttles and quotas “are applied on a best-effort basis and should be thought of as targets rather than guaranteed request ceilings,” and that clients may receive 429 responses they can resubmit “in a way that is rate limiting.” That is a documented behavior, not something you observed in your incident. Keep the two categories separate in the write-up. If the vendor documents a token-bucket model, say so and cite the page. If you are inferring the model from response patterns, say that instead.

Record your own request pattern with numbers

The vendor’s behavior is half the story. Your client’s behavior is the half you can change, and it needs numbers.

Capture the request rate your service was sending at the time of the incident, and the rate in the hour before. Note whether the traffic was steady or bursty, and what caused any change: a scheduled batch job, a retry storm, a traffic spike, a deploy that changed concurrency. If you have per-endpoint or per-tenant breakdowns, include them, because a single noisy tenant can consume a shared quota and the fix may be per-tenant limiting rather than a global change.

Then document the retry logic as it actually existed, not as you intended it to work. Specifically:

• Were retries bounded, and at what count or duration?
• Did retries use exponential backoff, and did they include jitter?
• Did the client honor any Retry-After header the vendor sent?
• Did retries run concurrently across workers, so that each worker’s backoff was independent and the aggregate rate stayed high?
• Did the retry logic distinguish a 429 from a 500, or did it treat all failures the same way?

Retries deserve their own subsection because they can turn a rate-limit incident into a self-inflicted one. If your client retried immediately on 429, the postmortem should say so and should show the request rate before and after retries began. That is an observation about your system, not a judgment about the engineer who wrote the retry loop.

Separate the trigger from the contributing conditions

A rate-limit incident rarely has a single root cause, and forcing one into the document produces a misleading record. Write down the trigger and the contributing conditions as separate lists.

The trigger is the event that started the incident: a traffic spike, a batch job that doubled the request rate, a vendor-side quota change, a retry storm. The contributing conditions are the properties of your system that let the trigger become an incident: no client-side limiter, no alert on 429 rate, no documented vendor quota in the runbook, no circuit breaker between your service and the dependency.

This split is what keeps the postmortem useful. “Vendor rate-limited us” is a trigger with no contributing conditions attached, and it produces no action items. “A batch job doubled our request rate; we had no client-side limiter and no alert on 429 rate” produces two fixes the team controls.

It also keeps the document blameless in the specific way that matters here. Google’s guidance is that a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.” A vendor is a team. The postmortem should describe what the vendor’s API did, with evidence, and should not characterize the vendor’s motives or competence. The same rule applies to the engineer who wrote the retry loop two years ago.

Label inferences about the vendor explicitly

You can observe the vendor’s responses. You cannot observe the vendor’s internal throttling state, its quota accounting, or why a particular request was rejected. When the postmortem needs to explain vendor behavior beyond what the response and documentation show, mark it as an inference.

A workable convention: put observed facts in the main text with a timestamp or a pasted payload, and put inferences in a clearly headed subsection such as “What we believe happened on the vendor side.” That subsection can say “we infer the quota is shared across our three API keys because all three began returning 429s within the same minute” without asserting it as fact. If a support ticket later confirms or contradicts the inference, update the postmortem and note the update date.

Record vendor communication and support tickets

If the team opened a support ticket, record the ticket number, the time it was opened, the time of the first response, and the resolution or current status. If the vendor’s status page was updated during the incident, note the timestamp of the update and what it said. If the team requested a quota increase, record the request date and the outcome.

This section is short but it prevents a common failure: six weeks later, nobody remembers whether the quota increase was approved, and the next incident repeats the discovery. If your contract includes service credits or SLA terms that the incident might qualify for, note the relevant clause and whether the team filed a claim. Do not estimate the credit amount in the postmortem; record the clause and the filing status, and let finance or the account owner handle the number.

Write action items with one owner and a tracking reference each

The action items are where the postmortem either changes something or becomes a document nobody reads again. Google’s SRE Workbook is direct about the failure mode: “Action items without clear owners are less likely to be resolved. It’s better to have a single owner and multiple collaborators.” The same chapter criticizes action items that use vague verbs like “improve” or “make better,” and notes that items tagged with equal priority give no guidance on what to do first.

For a rate-limit incident, split the action items into two groups.

Preventative items change the system so the incident is less likely or less severe:

• Add a client-side rate limiter with a configured ceiling below the vendor’s documented quota.
• Add exponential backoff with jitter to the retry path, and honor Retry-After when present.
• Add an alert on the 429 rate, or on the ratio of 429s to total requests, with a threshold tied to your observed baseline.
• Add a circuit breaker that stops sending requests when the 429 rate crosses a threshold, so the dependency can recover.
• Document the vendor’s quota and burst behavior in the runbook, with a link to the vendor’s documentation page.

Mitigative items shorten the next incident:

• Add a runbook entry describing how to identify a rate-limit incident, how to reduce request rate quickly, and how to open a vendor ticket.
• Add a dashboard panel showing request rate, 429 rate, and retry count side by side.
• Record the vendor’s support escalation path and the account identifier needed to open a ticket.

Each item gets one owner, a priority, and a tracking reference such as an issue number. If your team uses a postmortem template, the template should have columns for all three, because a table with an empty owner column is easier to notice than a paragraph that omits one.

What to leave out

Three things do not belong in this postmortem.

First, animated language. The SRE Workbook’s critique of a bad postmortem notes that “animated language and dramatic descriptions of events distract from the key message and erode psychological safety.” “The vendor completely failed us” and “the retry loop was obviously broken” are both animated, and both make the document harder to act on.

Second, claims about the vendor’s internal behavior that you cannot support. If the vendor’s documentation does not state a quota, do not write a number as if it were documented. Write what you observed and label the rest as inference.

Third, a single root cause when the evidence supports several contributing conditions. The trigger-plus-conditions structure is more accurate and more actionable than a one-line cause statement.

A constructed example

The following is an illustration, not a case record. A service calls a vendor API at roughly 50 requests per second with no client-side limiter. The vendor’s documentation states a steady-state limit of 40 requests per second and a burst of 100. A nightly batch job starts and doubles the request rate to about 100 per second. The vendor returns 429s with a JSON body containing a Retry-After header of 2 seconds. The client’s retry loop ignores the header and retries immediately, pushing the aggregate rate higher. The first 429 appears at 02:14; the on-call engineer is paged at 02:51 by an alert on queue depth, not on 429 rate.

The postmortem for that incident would record the pasted 429 payload and the Retry-After value, the request rate before and after the batch job started, the retry behavior including the ignored header, the detection gap of thirty-seven minutes, and action items for a client-side limiter, jittered backoff that honors Retry-After, and an alert on 429 rate. It would note that the vendor’s documented burst of 100 was exceeded and that the team inferred the batch job was the trigger, with the batch job’s start time as evidence. It would not say the vendor “throttled us unfairly” or that the engineer who wrote the retry loop “should have known.”

A checklist to paste into your template

The next time a third-party rate limit causes an incident, the postmortem should answer these questions:

• What did users experience, for how long, and which internal services were affected?
• When did the first rate-limit response appear, and when did a human first know?
• What was the exact status code, response body, and any retry headers, pasted verbatim?
• What does the vendor’s public documentation say about throttling, and what did we infer beyond it?
• What was our request rate before and during the incident, and what changed it?
• What did our retry logic actually do, including whether it honored Retry-After and whether it used jitter?
• What was the trigger, and what were the contributing conditions?
• What vendor communication happened, with ticket numbers and timestamps?
• Which action items are preventative, which are mitigative, and who owns each one?

If your team is still building the habit of writing recovery procedures down before an incident, the companion piece on writing the recovery checklist before you need it covers the runbook side of the same problem. The postmortem records what happened; the runbook is what the next on-call engineer reads at 02:14.

A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent recurrence, per Google’s SRE Book. The same chapter states the primary goals plainly: document the incident, understand the contributing causes, and — especially — put effective preventive actions in place. The document is not the deliverable. The closed, verified action item is.

For a team of two to fifteen engineers who are their own on-call rotation, the failure is rarely in the writing. It is in the handoff. During the incident, there was an incident commander, a timeline, and a resolution. The action item that replaces it enters a backlog with no owner, no date, and no definition of done, while feature work enters with all three. Losing that prioritization contest is the default, not a discipline failure.

The gap is structural, not cultural

Google’s SRE Workbook, in its postmortem culture chapter, walks through a deliberately bad postmortem and names the failure modes. The action items are mostly mitigative; the one “preventative” item suggests making humans less error-prone, which the chapter says is less reliable than changing automated systems and processes. Two items use vague language — “Improve” and “Make better” — which the chapter says makes success criteria hard to measure. Only one item was assigned a tracking bug, and the chapter notes that without a formal tracking process, action items from postmortems are often forgotten, resulting in outages.

The same chapter quotes Ben Treynor Sloss: “To our users, a postmortem without subsequent action is indistinguishable from no postmortem.” It also states that action items without clear owners are less likely to be resolved, and that a single owner with multiple collaborators beats shared ownership. That is the whole argument in two sentences. The rest is mechanics.

Atlassian’s incident postmortem guide makes the same point from the planning side: findings should feed back into the planning process so remediation work is prioritized alongside other work. It also says an unreviewed postmortem “might as well never have been written.” Review is necessary but not sufficient. An item can be reviewed, approved, and still rot.

Two failure modes that let items rot

Owner: “the platform team.” A team is not an owner. A team is a distribution list. When the item is due and nothing has happened, there is no name to ask. The Google Workbook’s guidance is explicit: one owner, multiple collaborators. The named owner does not have to do most of the work. The name exists so that escalation has an address.

Done: “merged.” A merged pull request proves code changed. It does not prove the change would have caught the incident. The distinction matters most for detection items — a new alert that fires in staging but pages nobody in production is merged and useless. Verification has to be chosen when the item is filed, while the person who wrote it still remembers what would have caught the failure.

The five-field action item

Every item needs five fields. If one is hard to fill in, the item is not ready to leave the postmortem meeting. This is a recommendation, not a published standard; it borrows the actionable/specific/bounded framing that PagerDuty’s postmortem writing guide attributes to Google’s Lunney, Lueder, and Beyer, and adds a verification field because “bounded” alone does not say how you will know it worked.

1. Classification. Guardrail (prevents the failure), detection (catches it sooner), process (changes how people respond), or knowledge (spreads what one person learned). The class determines what verification looks like.
2. One named owner. A person, not a team. Collaborators are listed separately.
3. Smallest shippable version. The version that could merge this sprint. A rate limit on one endpoint, not “redesign the throttling layer.” Large items get split, not deferred.
4. Verification method. How you will know it would have caught the incident: replay the trigger, inject the fault, or point at the alert that now fires. “Merged” is not verification.
5. Decision date. A calendar date written on the item the day it is filed. Default to the end of the next sprint for guardrail and detection changes. Anything further out needs the reason written beside the date.

PagerDuty’s postmortem template shows a lighter version of the same discipline: each action item is a JIRA ticket tagged with the incident date and severity so it can be found and reported later. The tag is the trigger. Without it, the item is a sentence in a document that nobody queries.

A review cadence that is not the incident process

The incident process ends when the postmortem is published. If follow-through depends on the next incident to prompt a review, the cadence is set by outages. That is the wrong clock.

For a small team, the cheapest structural fix is not new tooling. It is a recurring fifteen-minute review attached to a meeting that already exists — the weekly ops sync, the sprint planning opener, the Monday standup that already runs long. The agenda is one filtered list: open action items past their decision date, sorted oldest first, with the owner’s name on each row. A saved filter on the due-date field does this in Jira, GitHub Issues, or Linear without a new system.

Two rules keep the meeting short. Aging items go first, not newest. And an item that has slipped twice gets either a new date with a written reason or an explicit decision to drop it and accept the risk. Silent slippage is how a backlog becomes a graveyard.

Closure on evidence, not on a status field

An item closes when the verification method has been run and the result is recorded on the item: the replay output, the alert that fired, a link to the new test. Marking it done is not closing it. This is stricter than most small teams operate, and it is the step that prevents the “said it was done, nothing actually happened” failure.

For knowledge items — a runbook update, a decision record — verification is a review by someone who was not on the incident. If the runbook cannot be followed by a teammate who was not there, it is not done. This connects directly to the practice of writing the recovery checklist before you need it; the Recovery Checklist article covers the document side of the same problem.

A quarterly recurrence review

Once a quarter, list the incident classes from the previous four quarters and mark each one recurred or did not. A recurrence means the fix was wrong or never shipped. No recurrence tells you which category of fix is worth repeating. This is the feedback loop on the action-item process itself, and it is the only way to learn whether your guardrails or your detection changes are actually buying reliability.

The Google Workbook’s rack-decommission case study is a useful reference point here. Three years after the original outage, a similar incident occurred, and the chapter states that the action items implemented from the original postmortem dramatically reduced the blast radius and rate of the second incident. That is what a shipped, verified fix looks like from the outside: the same class of failure returns smaller.

A worked hypothetical for a small team

The following is a constructed example, not a real incident record. It illustrates how the five fields change what happens after the third postmortem.

A six-person platform team runs a webhook delivery queue. It overflows three times in a year. Each incident gets a competent postmortem. The first two produce action items like “improve queue monitoring” and “add backpressure,” owned by “the platform team,” with no date. Neither ships. The third postmortem opens with an engineer copying from the first.

Under the five-field format, the third postmortem produces three items instead:

• Detection: alert when queue depth exceeds 80% of the worker pool’s drain rate for five minutes. Owner: one named engineer. Verification: replay the prior incident’s traffic pattern in staging and confirm the alert fires. Decision date: end of next sprint.
• Guardrail: per-tenant rate limit on webhook enqueue. Owner: a second named engineer. Verification: load test at 3x peak with one tenant saturated; confirm other tenants’ delivery latency stays within SLO. Decision date: two sprints out, with the reason written beside it.
• Knowledge: runbook section for queue saturation, including the drain command and the escalation path. Owner: the on-call engineer who worked the incident. Verification: a teammate who was not on the incident follows the runbook in staging without help. Decision date: one week.

The difference is not effort. It is that each item now has an address, a date, and a test that can fail. The weekly review surfaces the rate-limit item when it slips. The quarterly recurrence review asks whether the queue class returned after the alert shipped. If it did, the detection fix was the wrong category, and the next postmortem should say so.

What this costs a lean team

Fifteen minutes a week, one saved filter, and the discipline to refuse an action item that has no owner or no verification. The tradeoff is real: some items will be dropped rather than carried indefinitely, and dropping them is an explicit risk decision that should be recorded. That is cheaper than a backlog of items nobody believes will ship, and it is the only version of follow-through that survives a team with no dedicated reliability staff.

FAQ

What should a postmortem action item include?

A classification, one named owner, the smallest shippable version of the fix, a verification method, and a decision date. An item missing an owner or a date is a suggestion, not an action item.

Why do postmortem action items never get done?

Because the incident process ends when the document is published, and implementation then depends on backlog prioritization, where action items rarely win against feature work. The fix is structural: a recurring review outside the incident process, aging surfaced first, and closure on verification evidence.

Doesn’t a named owner conflict with blameless postmortems?

No. Blamelessness is about how the incident is analyzed — focusing on contributing causes rather than indicting individuals. Ownership is about who carries the follow-up work. The Google Workbook treats these as separate concerns and recommends a single owner with collaborators for each action item.

Do we need a new tool?

Usually not. A saved filter on the due-date field in whatever tracker you already use is enough to run the weekly review. The failure mode is not tooling; it is the absence of a cadence and the absence of a verification field.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

The shirt reads I WEAR THIS SHIRT PERIODICALLY, and beneath that line sit four periodic-table-style tiles: 53 I, 74 W, 22 Ti, 16 S. The tile symbols are I, W, Ti, S. They are not a literal spelling of the sentence above them. That gap is the joke, and it is also the most interesting thing about the design.

Most slogan shirts ask you to read a sentence and agree or laugh. This one asks you to read a sentence, then read a second, smaller system underneath it, then notice that the two do not match. The tiles are not an abbreviation of the words. They are a chemical spelling of a different phrase — I W Ti S — that happens to sound like I wear this when you say the element symbols aloud. The word periodically then lands as a separate double meaning: it refers to the periodic table that supplies the tiles, and to the frequency with which the wearer puts the shirt on. The design works because the viewer has to do the conversion. The shirt is not a billboard. It is a small puzzle with a punchline.

What the tiles actually say

Read literally, the four tiles are:

• 53 I — iodine
• 74 W — tungsten
• 22 Ti — titanium
• 16 S — sulfur

The symbols spell I W Ti S. Say that out loud and you get something close to I wear this — a compressed, element-flavored version of the sentence printed above. The word periodically then lands as a double meaning: it refers to the periodic table that supplies the tiles, and to the frequency with which the wearer puts the shirt on. The design is doing two things at once, and it is doing them with the same four tiles.

That is a more disciplined piece of graphic design than most pun shirts manage. The tiles are not decoration. They are the mechanism. Remove them and the sentence is just a mild boast. Keep them and the sentence becomes a small act of translation.

Why the mismatch matters

The common failure mode in novelty design is over-explanation. A shirt prints the joke, then prints the explanation of the joke, then prints a small arrow pointing at the explanation. The viewer is not asked to do anything. The shirt is a label.

This design avoids that. It prints the sentence and the tiles and stops. The viewer who knows the periodic table gets the conversion immediately. The viewer who does not gets a slightly mysterious set of numbered boxes and a sentence that almost explains them. Either way, the shirt does not condescend. It assumes the wearer is the kind of person who will enjoy the gap between the two readings.

That assumption is the same one we make about our readers. A runbook that explains every command is a tutorial. A runbook that names the failure mode and points at the recovery step assumes a competent operator who will read the rest. The shirt is closer to the second kind of document. It gives you the pieces and trusts you to assemble them.

The difference between a slogan and a usable promise

There is a useful distinction here for anyone who writes internal documentation, on-call handoffs, or team charters. A slogan is a sentence that sounds true. A usable promise is a sentence that tells you what will happen next. The shirt’s top line — I wear this shirt periodically — is a slogan. The tiles underneath are the promise: here is the actual mechanism, here is what the symbols are, here is the conversion you need to perform.

Most team charters fail the same way. They print the slogan — we value reliability, we own our incidents — and never print the tiles. The reader is left with a sentence that sounds true and no way to check it. The shirt, whatever else it is, does not make that mistake. It shows its work.

We keep a small decision log for exactly this reason. When we write down a policy, we try to write down the mechanism next to it: who triggers it, what command runs, what the rollback is. The slogan goes in the title. The tiles go in the body. If we cannot fill in the tiles, the slogan is not ready to ship.

What the shirt is not

It is worth being clear about what this design does not claim. It does not claim the wearer is a chemist. It does not claim the shirt improves anything, cures anything, or signals membership in a profession. It is a pun about the periodic table and the word periodically, printed on a shirt. The tiles are a visual mechanism, not a credential.

That restraint is part of why it works. The shirt is not trying to be a personality test. It is trying to be a small, well-made joke that rewards a second look. The viewer who reads the tiles and says oh, I W Ti S has done exactly the work the design asked for. The viewer who does not still gets a sentence that makes sense on its own.

Where the design sits in the wardrobe

We have written before about shirts that function as boundary markers — the Not Friendly Do Not Touch shirt, the meh shirt, the composition notebook shirt. Those designs work because they say something about the wearer’s relationship to attention, effort, or interruption. This one is different. It is not a boundary. It is an invitation to a small piece of wordplay, worn in the same spirit as a band shirt or a conference shirt: a signal that the wearer enjoys a particular kind of puzzle.

That makes it a reasonable off-duty object for technical readers, not because it demonstrates engineering competence, but because it rewards the same habit — reading a system closely, noticing that the labels do not match the sentence, and enjoying the conversion. The shirt is available on Amazon at the Quirky Cotton Collective listing if you want to read the tiles yourself.

What we’d do differently

If we were designing a shirt like this, we would resist the urge to add a fifth tile or a caption explaining the pun. The design works because it stops at four tiles and one sentence. The temptation in our own documentation is the same: to add one more clarifying paragraph, one more diagram, one more note that explains the note. The shirt is a reminder that the gap between the sentence and the tiles is the point. Fill it in and the joke disappears. We would leave the gap.

The first thing we notice is the triangle. A yellow caution triangle, the kind that usually means wet floor or high voltage, sits above three lines of block lettering: NOT FRIENDLY, DO NOT TOUCH. Below the words, a hand is crossed out — a simple pictogram of a palm with a diagonal slash through it. The whole composition is borrowed from industrial signage, the visual grammar of hazard warnings, and it is doing something slightly different here. It is not warning you about a machine. It is warning you about a person.

That shift — from equipment to human — is the entire joke, and also the entire argument. The shirt is not a slogan in the usual sense. A slogan makes a claim about the wearer’s mood or taste. This is closer to a boundary marker: a terse, legible statement of access policy, printed in the one visual language most people already know how to read without thinking. We keep coming back to that distinction because it matters for how the object works in practice. A shirt that says not friendly in plain text is a joke. A shirt that says it inside a caution triangle, with a crossed-out hand, is a joke that has been formatted as an instruction.

What the sign language actually does

Industrial signage is designed to be parsed at a glance, by people who are not paying attention, under conditions where a mistake has consequences. The caution triangle, the crossed-out hand, the all-caps imperative — these are not decorative choices. They are a compressed protocol. When you put that protocol on a t-shirt, you are borrowing its authority. The wearer is not just saying leave me alone; they are saying it in the register of a system that expects to be obeyed.

We think that is why the design reads as more than a novelty. The crossed-out hand is the key element. It is not a hand waving, not a handshake, not a hand held up in a stop gesture. It is a hand with a slash through it — the universal symbol for this action is prohibited. The shirt is not describing a personality trait. It is prohibiting a behavior. That is a meaningful difference, and it is the difference between a shirt you wear to be funny and a shirt you wear to be left alone.

We should be careful here. We are reading the artwork, not making a claim about the wearer’s actual social preferences. The design is a visual argument, not a diagnostic tool. But the argument it makes is coherent: it takes the grammar of a hazard sign and applies it to the hazard of unwanted interaction. Whether that lands as funny, rude, or simply accurate depends entirely on the room.

The difference between a slogan and a usable promise

In our day jobs we spend a lot of time thinking about the difference between a statement and a procedure. A runbook that says restore from backup is a slogan. A runbook that says run this command, check this output, escalate to this person if it fails is a usable promise. The gap between those two things is where most operational failures live.

The Not Friendly Do Not Touch design sits on the usable-promise side of that gap, at least visually. It does not ask the reader to infer the wearer’s boundaries from context. It states them in the most legible format available. The caution triangle is a trigger. The crossed-out hand is the action. The lettering is the policy. It is, in its own small way, a well-designed interface: the reader knows exactly what is expected of them within about half a second of looking at it.

That is not nothing. Most social signals are ambiguous by design. This one is not. Whether you find that refreshing or abrasive probably depends on how much of your day you spend explaining boundaries that you wish were already obvious.

Why the warning-sign format is the right container

We have written before about shirts that use diagram language — the ramp-and-ball physics pun, the composition notebook surface — and the common thread is that the visual format does most of the work. A joke printed in plain text has to be read. A joke printed as a diagram has to be recognized. Recognition is faster and, for a certain kind of wearer, more satisfying.

The warning-sign format is particularly well suited to this message because it carries an implicit this is not personal clause. A hazard sign is not angry at you. It is not judging you. It is simply informing you that a condition exists and that a behavior is prohibited. When the wearer puts that format on, they are borrowing that same impersonal tone. The shirt is not saying I dislike you. It is saying this area is restricted. That is a much easier thing to wear in public, and a much easier thing to laugh at, because it removes the sting of rejection from the message.

We think that is the smartest thing about the design. It takes a boundary that could read as hostile and formats it as infrastructure. The crossed-out hand is not a rejection of a specific person. It is a policy. Policies are easier to accept than insults.

What the shirt is not

It is not a claim about introversion as a clinical category, and we would not read it as one. The retail title uses the word introvert, but the artwork itself does not. The artwork says not friendly and do not touch, which are behavioral instructions, not personality labels. We mention this because it is easy to collapse the two, and the collapse does the design a disservice. The shirt is funnier and more useful as a boundary marker than as a personality badge.

It is also not a substitute for actually telling people what you need. A shirt is a low-stakes signal, not a substitute for a conversation. We would not want to overstate its power. But as a low-stakes signal, it is unusually well constructed: legible, consistent, and formatted in a visual language that most people already know how to read.

Where it fits

We keep a small decision log for objects like this — things that are not tools but that shape how a team interacts. A shirt is not a runbook. But the question it raises is the same one we ask about runbooks: does the reader know what to do next? The Not Friendly Do Not Touch design answers that question clearly. The reader knows exactly what to do next. They know not to touch. They know the wearer is not friendly. The interface is complete.

If you want to look at the listing, it is here: Not Friendly Do Not Touch Funny Warning Sign Introvert. We are not going to pretend we tested it. We read the artwork, and the artwork is doing something specific and coherent. That is enough for us to recommend it as a well-designed object, which is a different thing from a recommendation to buy it.

What we’d do differently

If we were designing a shirt like this, we would probably be tempted to add more text — a second line, a clarification, a joke that undercuts the boundary. We would be wrong. The strength of this design is its restraint. The caution triangle, the crossed-out hand, the three words. Nothing else. The temptation to explain is the enemy of a usable signal, and this shirt resists it. We would do well to remember that the next time we are writing a runbook and reach for a fourth paragraph of context. Sometimes the crossed-out hand is enough.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

On a dark shirt, centered, in very small lowercase letters, the word meh. appears. No animal illustration. No pride variant. No diagram, no arrow, no punchline in a second line. The period is part of the mark. The scale is the argument: the word is set small enough that a reader has to be close to parse it, which means the shirt does not perform for a room. It performs for one person at conversational distance, if it performs at all.

That is a different object from the slogan shirt as it usually works. Most slogan shirts are legible across a parking lot. They are designed to be read by strangers, which makes them a kind of broadcast. This one is designed to be read by someone already close enough to notice. The difference matters for anyone who has spent time thinking about how teams communicate under stress, because it is the same difference between a status page and a runbook.

We keep a small decision log for our own systems, and one recurring entry is about audience. A status page is written for people who are not us: customers, support, whoever is watching the graph. A runbook is written for the person who will be paged at 3 a.m. and needs the next command, not a narrative. Both are text. Both are terse. But they are calibrated to different distances, and confusing them produces the specific failure where a status page reads like an internal note or a runbook reads like a press release.

The meh. shirt is calibrated to the close distance. It is not a status page. It is not trying to inform a crowd that something is wrong. It is closer to a runbook entry that says only what the operator needs and nothing more: a single word, a period, done. The period is doing real work. Without it, meh is a sound. With it, meh. is a verdict. The punctuation turns an interjection into a statement, and the small type keeps the statement from becoming a speech.

I want to be careful about what this observation supports. It supports a reading of the artwork. It does not support a claim that the shirt improves anything, communicates a policy, or represents a movement. It is a garment with a word on it. The interesting question is not whether it is deep. The interesting question is what kind of message survives being set this small, and why a designer would choose that constraint.

Small type is a filter. It selects for proximity and attention. In a work context, the messages that survive that filter tend to be the ones that do not need volume: a boundary, a handoff, a single line in a channel that says the thing and stops. The messages that fail the filter are the ones that need an audience to complete them. A joke that requires a laugh track is not the same object as a joke that lands in a quiet room. The meh. shirt is built for the quiet room.

There is a useful distinction here between a slogan and a usable promise. A usable promise tells you what will happen next. It has an owner, a trigger, and a condition. A slogan tells you what someone feels or claims. The meh. shirt is a slogan, and I think that is the correct category for it. It does not promise uptime, recovery, or anything operational. It does not pretend to be a runbook. It is a mood rendered as typography, and the restraint is what keeps it honest. A shirt that tried to be a usable promise would need more words, and more words would make it worse.

This is where the object connects to a habit I think technical teams should steal from designers: decide the distance before you decide the words. If the message is for the person already in the room, you can be terse. If it is for the person who has not arrived yet, you need context. Most communication failures I have watched are distance failures, not tone failures. Someone writes a close-distance message and broadcasts it, or writes a broadcast message and wonders why the on-call engineer ignores it. The fix is rarely more empathy. It is usually a smaller audience and a shorter sentence.

The shirt is available as a standard t-shirt through the listing at https://www.amazon.com/dp/B0FXC6KKSR. I am describing the artwork as observed in the native garment preview, not a wearing experience. I have not tested the fabric, and I am not going to pretend that a shirt is a reliability practice. It is an object with a word on it, and the word is set small on purpose.

What I keep returning to is the period. In a runbook, the period is the end of a step. It means the command is complete and the next thing can happen. In the shirt, the period does something similar: it closes the word so it does not trail off into a question or a shrug. meh. is not ambivalence. It is a decision to stop. That is a small, repeatable move, and it is one I would like to see more of in incident communication: say the thing, end the sentence, wait for the next signal.

What we’d do differently

If we were writing the listing copy, we would resist the urge to explain the joke. The artwork is already doing the work, and explanation would only add distance. We would also avoid claiming the shirt is for a particular personality type, because the design does not establish that. The observation is the type: small, centered, lowercase, dark ground, period included. Everything else is the reader’s business.

A bus factor audit is not a documentation project. It is a mapping exercise. The deliverable is a per-service list of people who can actually restart the service, plus the evidence behind each name. If the list has names but no evidence, you have a hypothesis, not a map.

For a team of 2–15 engineers who are their own on-call rotation, the audit has one job: find the services where the answer to “who can restart this?” is a single person, a stale runbook, or a name that has never done it under pressure. Then turn those gaps into drills, shadowing, and runbook work before the next incident finds them for you.

Why this is a mapping exercise, not a documentation project

Documentation projects tend to produce artifacts that describe how a service works. A bus factor audit produces a different artifact: a table that says who can perform a specific recovery action and how you know. The distinction matters because a runbook can be complete and still leave you with a bus factor of one. The runbook is not the capability. The person who has executed the runbook is the capability.

PagerDuty’s incident response guidance notes that knowledge of how to handle incidents is often lost when a team member leaves, which is why a documented process becomes more crucial over time. That is the failure mode the audit is designed to surface before it becomes an incident. The audit does not replace the documentation; it tells you where the documentation is the only thing standing between you and an unowned service.

NIST SP 800-34 Rev. 1 frames contingency planning as a process of evaluating information systems and operations to determine requirements and priorities. A bus factor audit is a narrow, repeatable version of that evaluation, scoped to restart and recovery actions rather than the full contingency plan.

Define the unit of audit: service, restart action, evidence

The unit of work is not the service as a whole. It is the restart action. A single service may have several restart paths, and different people may be able to perform each one:

• Process restart on the primary host or container.
• Failover to a standby or secondary region.
• Restore from backup (database, object store, configuration).
• Rebuild from infrastructure-as-code when the host is gone.
• Break-glass access path when normal credentials are unavailable.

For each path, record the evidence type. Three levels are enough for most lean teams:

• Has done it in an incident. The strongest evidence. The person executed the action under real pressure, with real consequences.
• Has done it in a drill. Strong evidence. The person executed the action in a controlled setting, ideally with a time constraint and a defined success condition.
• Has read the runbook. Weak evidence. This is a starting point, not a qualification.

Google’s SRE workbook describes a pre-on-call checklist that includes administering production jobs, rolling back a bad software push, bringing up additional serving capacity, and using monitoring systems. That checklist is a useful model for the evidence bar: the question is not “have you read about this?” but “have you practiced this?”

Build the map

One row per service. Columns for primary, backup, evidence type, last verified date, and known gaps. Keep it in a place the team already looks at, not a separate tool that will be forgotten.

The table is deliberately small. For a 2–15 engineer team, the audit should be completable in a few sessions. If it is taking longer, the scope is too broad. Start with the services that have the highest blast radius and the fewest people.

Use the map to find single points of failure

Three patterns are worth flagging explicitly:

• One name, no backup. The service has a primary but no second person who has executed the restart path. This is the classic bus factor of one.
• Names but no evidence. The service has a primary and a backup, but both are at “has read the runbook.” This is a bus factor of one with extra steps.
• The runbook is the only artifact. No one on the current team has executed the restart path, and the runbook was written by someone who has left. This is the highest-risk category because the runbook may be stale in ways no one can see.

Google’s SRE workbook notes that playbook details go out of date at the same rate as production environment changes, and that for daily releases, playbooks might need an update on any given day. A runbook that has not been executed by a current team member is a runbook whose staleness is unknown.

Turn gaps into drills

The audit’s output should feed directly into drills and shadowing, not into a separate documentation backlog. The goal is to move people from “has read the runbook” to “has done it in a drill.”

Google’s SRE workbook describes the Wheel of Misfortune as a training exercise where teams role-play recent incidents to practice debugging production problems. A bus factor drill is a narrower version: pick a service with a gap, pick a person who has not executed the restart path, and run the restart in a controlled setting. The success condition is not “the service came back” but “the person can describe what they did and why.”

For restore paths, a restore lottery is a useful format. One person is selected at random, given the backup and the runbook, and asked to restore to a point in time. The exercise surfaces both the person’s capability and the runbook’s accuracy. If the restore fails because the runbook is wrong, that is a finding, not a failure.

For services with no second person, shadowing is the first step. The primary executes the restart path while the backup observes and asks questions. The next step is the reverse: the backup executes while the primary observes. Only after that does the backup count as evidence.

Keep the map alive

A bus factor map that is not reviewed becomes a stale spreadsheet. Tie the review to artifacts the team already uses:

• On-call handoffs. When the rotation changes, check whether the new on-call has executed the restart paths for the services they are now responsible for.
• Playbook updates. When a runbook changes, mark the corresponding map row as unverified until someone executes the new path.
• Postmortem action items. When a postmortem identifies a recovery gap, add it to the map with a review date.

The Google SRE book states that an unreviewed postmortem might as well never have existed, and that regular review sessions are encouraged to ensure each completed draft is reviewed. The same logic applies to the bus factor map: an unreviewed map is a map you cannot trust.

Google’s SRE book lists common postmortem triggers including user-visible downtime or degradation beyond a certain threshold, data loss of any kind, on-call engineer intervention, a resolution time above some threshold, and a monitoring failure. Any of these triggers is also a signal to re-check the map for the affected service.

What to do when the map shows a gap you cannot close this quarter

Not every gap can be closed immediately. A team of five engineers cannot train everyone on everything in a single quarter. When a gap cannot be closed, record it explicitly:

• The service and restart path.
• The current evidence level.
• The accepted risk, in plain language.
• A review date.

The alternative is to leave the service silently unowned, which is the condition the audit was meant to find. An accepted risk with a review date is a decision. An unrecorded gap is an accident waiting for a trigger.

For teams running on AWS, GCP, or bare metal, the audit is the same exercise. The AWS Well-Architected Framework reliability pillar notes that achieving reliability can be challenging in traditional on-premises environments due to single points of failure, lack of automation, and lack of elasticity. The bus factor audit is a way to find the human single points of failure that architecture diagrams do not show.

A short, repeatable audit cadence

For a 2–15 engineer team, a quarterly cadence is enough to keep the map current without turning it into a project. The cadence can be as simple as:

1. Pick the three services with the highest blast radius.
2. For each, list the restart paths and the people who can execute them.
3. Mark the evidence level for each person.
4. Identify one gap per service and schedule a drill or shadowing session.
5. Update the map with the drill result and a new review date.

The audit is not a one-time report. It is a working artifact that changes as the team changes. The goal is not to eliminate every gap. The goal is to know where the gaps are, so that when the pager fires at 03:00, the person who answers has a realistic path to recovery.

Frequently asked questions

How is a bus factor audit different from a runbook review?

A runbook review checks whether the documentation is accurate. A bus factor audit checks whether a person can execute the documented steps. A runbook can pass review and still leave you with a bus factor of one if no one on the current team has executed it.

How many people need to be able to restart a service?

There is no universal number. The audit’s job is to make the current state visible, not to enforce a threshold. For a team of 2–15 engineers, the practical question is whether the service has at least one person who can execute the restart path when the primary is unavailable, and whether that person has evidence beyond reading the runbook.

What if the only person who can restart a service is leaving?

That is the highest-priority gap the audit can find. The response is a drill or shadowing session before the person leaves, with the goal of moving at least one other person to “has done it in a drill.” If that is not possible, the gap should be recorded as an accepted risk with a review date and a plan for the next quarter.

Should the audit include break-glass access?

Yes. Break-glass access is a restart path. If the normal credentials are unavailable, the break-glass path is the one that matters. The audit should record who can use it and whether they have tested it.

How often should the map be reviewed?

Quarterly is a reasonable default for a lean team. The map should also be reviewed after any incident that required a restart, after any on-call rotation change, and after any playbook update that changes a restart path.

For a related practice, see Write the Recovery Checklist Before You Need It, which covers the checklist side of the same problem: what to have ready before the restart is needed.

Most handover notes fail the same way. The author writes a careful description of the backup system — where the repository lives, which tool runs nightly, roughly how a restore works — and the reader, three days into a vacation, discovers the note describes intent rather than a path. The repository is reachable. The command is almost right. The backup set the reader picked cannot actually be restored to the target they need, and nothing in the note said so.

The unit of value in a handover note is not the description of the backup system. It is the executable recovery path: the exact command, the inputs it needs, and the observable that proves it worked. If the substitute cannot run it, the note has failed regardless of how complete it reads.

This is a craft note for lean teams — two to fifteen engineers, no dedicated SRE department, one on-call rotation — about what to write down before the person who can restore the database becomes unreachable for two weeks. It is not a template to fill in. It is a set of decisions about what belongs in the note and what does not.

Start from the failure mode, not the format

The failure mode is specific. The author leaves. A restore is needed. The substitute opens the note, finds a paragraph explaining that PostgreSQL maintains a write ahead log in pg_wal/ that records every change to the database’s data files, and finds a command that looks plausible. They run it. It completes. They start PostgreSQL. It comes up. The table they needed is still missing, because the backup they restored ended after the data was lost, and PostgreSQL can only play forward, not backward.

That is not a hypothetical. The pgBackRest user guide documents exactly this scenario in its point-in-time recovery walkthrough: if the wrong backup is selected, recovery to the required time target fails, and the log output does not make the failure obvious. The key is to look for the presence of the “recovery stopping before…” and “last completed transaction…” log messages. If they are not present, the recovery to the specified point-in-time was not successful.

A note that does not tell the reader to check for those log lines has not equipped them to know whether they succeeded. A note that does not tell them which backup sets are valid together has not equipped them to choose.

What the note must contain for each recovery scenario

For each scenario the team actually cares about — full cluster restore, point-in-time recovery to a timestamp, single-database restore, file-level restore from the object store — the note needs four things:

1. The exact command, with the flags that matter, not a paraphrase.
2. The inputs the command needs: which repository, which backup set, which target, which key material, which host.
3. The dependency chain: what else must be valid for this command to produce a usable result.
4. The observable that proves it worked — a log line, a query result, a checksum, a row count.

Anything else is context. Context is useful, but it is not the note. The note is the four items above, written so the reader’s first hour is spent executing rather than interpreting.

The PostgreSQL and PITR path

Continuous archiving combines a file-system-level backup with backup of the WAL files. Recovery restores the file system backup and then replays the backed-up WAL files to bring the system to a current state. That is the mechanism. The note’s job is to make the mechanism’s prerequisites explicit.

The first prerequisite is a continuous sequence of archived WAL files extending back at least as far as the start time of the backup. Not “some WAL files.” A continuous sequence, back to the backup’s start. If there is a gap — an archiving failure that was not noticed, a repository that filled, a version mismatch that stopped archiving — the backup is not recoverable to a point after the gap. The note should say where to check for gaps and what a gap looks like.

The second prerequisite is that pg_dump and pg_dumpall are not part of this path. They do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. If the team has both a logical dump and a physical backup, the note should say which one is for which purpose, because a substitute under pressure will reach for the one they recognize.

The third is that WAL archiving does not restore changes made to configuration files — postgresql.conf, pg_hba.conf, pg_ident.conf. Those are edited manually rather than through SQL operations. The note should say where the current configuration lives and how it is restored, because a cluster that comes up with the wrong pg_hba.conf is a cluster that comes up but does not accept the connections the application needs.

The fourth is the disk-space failure mode. If the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until space is freed. The note should say which filesystem to check and what the safe threshold is, because the substitute may be the person who has to free space at 2 a.m.

The pgBackRest path

pgBackRest’s restore command defaults to selecting the latest backup from the first repository where backups exist. That default is convenient and dangerous. The note should state it, because a reader who assumes the tool will pick the right backup will not pass --set or --repo, and the default may not be the backup they need.

For point-in-time recovery of --type=time or --type=lsn, pgBackRest selects a backup whose stop time is before the target time, or whose stop LSN is less than or equal to the target LSN. If no matching backup is found, it raises an error for both types; it does not fall back to the latest backup. That behavior belongs in the note verbatim, because the failure mode is a reader who assumes a fallback exists and does not check the error.

The dependency chain is the other thing the note must state. A differential backup requires both the differential backup and the previous full backup to be valid in order to restore. An incremental backup requires all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup to all be valid. The note should name which sets are valid together for the scenarios the team cares about, rather than pointing at “the backup.”

Two operational facts belong in the note as well. pgBackRest encrypts the repository based on a user-provided password, so the password is part of the recovery path, not a separate concern. And the local and remote pgBackRest versions must match exactly; if there is a mismatch, WAL archiving and backups will not function until the versions match. A note that does not record the version in use at the time of writing will be wrong after the next upgrade.

The restic and age path

For file-level backups, the note’s job is to separate two claims that are easy to conflate: “the repository exists” and “this person can decrypt and restore it.” Only the second one matters during a vacation.

age is a file encryption tool, format, and Go library with small explicit keys, post-quantum support, no config options, and UNIX-style composability. Files can be encrypted to multiple recipients by repeating the recipient flag, and every recipient will be able to decrypt the file. Identity files contain one or more secret keys, one per line, or an SSH key.

The note should record: where the repository lives, which recipients the backups are encrypted to, where the identity files are stored, and how the substitute obtains access to them. If the identity file is itself passphrase-protected and stored remotely, the note should say so, because that is a second access step that a substitute will not discover until they need it.

The note should also record what a successful restore looks like. For a restic repository, that is a restic check result and a test restore of a known file. For an age-encrypted archive, that is a decryption that produces the expected file. The observable is the point.

Break-glass access and offboarding hygiene

The handover note is where access hygiene and recovery access meet, and the tension is real. Least privilege says the substitute should not hold standing credentials they do not need. Recovery says the substitute must be able to reach the repository and the key material when the author is unreachable.

The resolution is to write down the break-glass path rather than to grant standing access. The note should name: the mechanism by which the substitute obtains elevated access, who approves it, how long it lasts, and how it is revoked. It should name the offboarding step that removes the author’s access when they return, so the break-glass grant does not become permanent. And it should name the person who can approve the grant when the author is the one who normally would.

This is the part of the note most likely to be skipped, because it is administrative rather than technical. It is also the part most likely to be needed, because a substitute who cannot reach the repository cannot run any of the commands the rest of the note describes.

The pre-vacation drill

The note’s only real test is running it as the substitute would. Not reading it. Running it.

The author picks a scratch target — a throwaway instance, a staging cluster, a container — and executes the note’s commands in the order the note presents them, using only the note as a guide. Every place the author has to supply knowledge that is not in the note is a gap. Every command that fails because a prerequisite was not stated is a gap. Every observable the author checks without the note saying to check it is a gap.

The record of that drill is the note’s revision history. It is more useful than a version number, because it says what was tested and when. A note that was drilled against pgBackRest 2.59.3 and PostgreSQL 17 is a different artifact from a note that was written against an earlier version and never re-run.

The drill also surfaces the failure behavior the note needs to record. A substitute who has run the commands once, on a scratch target, with the author watching, is a substitute who will recognize the difference between a restore that succeeded and a restore that completed.

Review, ownership, and re-verification

The note should have a named reviewer and a date. The reviewer is not a second author; they are the person who reads the note as a substitute would and reports what they could not follow. The date is the point from which re-verification is due.

The re-verification trigger is tied to change, not to a calendar. A note that was true at one pgBackRest version may be false at the next, because the local and remote versions must match exactly. A note that was true before a topology change may point at a repository that no longer exists. A note that was true before a key rotation may reference an identity file that has been replaced. The note should say what changes invalidate it, so the team knows when to re-drill.

This is the same discipline the SRE literature applies to postmortems. A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring. An unreviewed postmortem might as well never have existed. The same is true of a handover note. An unreviewed note is a document, not a recovery path.

It is worth defining, before the vacation rather than during it, what triggers a note review. The SRE book’s guidance on postmortem criteria applies here: define the criteria before the event so everyone knows when the process is necessary. For a handover note, the criteria are simpler — a version change, a topology change, a key rotation, or a drill that failed.

What the note is not

The note is not a postmortem. A postmortem explains what happened and why; a handover note explains what to do next. The two share a discipline of specificity and a refusal to blame, but they serve different readers at different times.

The note is not a runbook for every service. It is the recovery path for the things the author uniquely knows how to restore. If the team already has a runbook that a substitute can follow, the note should link to it rather than duplicate it. The Recovery Checklist Before You Need It is a useful companion for the general shape of a recovery checklist; the handover note is the specific, dated, drilled instance of it for one person’s absence.

The note is not a knowledge dump. A note that explains the history of the backup system, the reasons for each architectural choice, and the tradeoffs considered is a design document. Those are valuable. They are not what a substitute reads at 2 a.m. The note should be short enough to read in the first hour and specific enough to execute in the second.

Frequently asked questions

How long should the handover note be?

Long enough to contain the executable recovery paths for the scenarios the team cares about, and no longer. For a single PostgreSQL cluster with pgBackRest and a restic repository, that is typically a few pages: one section per scenario, each with the command, the inputs, the dependency chain, and the observable. Length is not the measure. The measure is whether a substitute can run it without asking the author a question.

Should the note include the backup password or key material?

No. The note should say where the key material is stored and how the substitute obtains access to it, not contain the material itself. pgBackRest encrypts the repository based on a user-provided password, and age identity files contain secret keys; both belong in a secrets manager or an equivalent access-controlled store, with the note recording the path and the approval step. A note that contains the key is a note that leaks when it is shared.

What if the substitute has never run a restore before?

Then the pre-vacation drill is not optional. The author runs the note’s commands as the substitute would, on a scratch target, and records what broke. The substitute should run them too, with the author available but not intervening. A note that has only been read is a hypothesis; a note that has been executed is a procedure.

How often should the note be re-verified?

Tie re-verification to change rather than to a calendar. A pgBackRest version change, a PostgreSQL major-version upgrade, a repository migration, a key rotation, or a topology change all invalidate parts of the note. The note should name those triggers so the team knows when to re-drill. A calendar-based cadence is a fallback for teams that cannot detect the changes, not the primary mechanism.

Does the note replace a postmortem when a restore fails?

No. If a restore fails during the author’s absence, that is an incident, and it merits a postmortem. The SRE book’s guidance is that postmortems are expected after any significant undesirable event, and that a blameless postmortem focuses on identifying the contributing causes without indicting any individual or team. The handover note is what the team uses during the incident; the postmortem is what the team uses afterward to make the next note better.

What is the single most important thing to write down?

The observable that proves the restore worked. Commands can be copied from documentation. The dependency chain can be reconstructed from the tool’s behavior. But the observable — the log line, the query result, the checksum — is the thing that tells the substitute whether they are done or whether they have produced a successful-looking but useless result. Write that down first, and write it for every scenario.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

Start with the rectangle. On the Composition Notebook Costume Teacher Group Halloween shirt, the printed artwork is a black-and-white speckled notebook-cover rectangle with a dark binding strip down the left side. Inside that rectangle sits a white ruled label headed COMPOSITION, and inside the label, an outlined ghost and an outlined pumpkin. The shirt is a printed notebook cover, not a physical notebook. That distinction matters more than it first appears.

We keep a small decision log at Gray Haven Lab, and one recurring entry is about surfaces. A runbook is a surface. A status page is a surface. A notebook cover is a surface too, but a strange one: it is the part of the object you see when the object is closed. The printed design on this shirt reproduces the closed state. It does not show pages, handwriting, or a spine cracked from use. It shows the cover as a graphic, which means the joke — if it is a joke — is about recognition rather than use.

That is the useful argument here. A composition notebook cover is one of the few pieces of school stationery that functions as a label before it functions as a tool. The speckled black-and-white pattern, the dark binding strip, and the ruled label are not decorative choices; they are the visual grammar of a mass-produced object that generations of students and teachers can identify from across a room. When that grammar is transferred to a shirt, the wearer becomes the closed object. The ghost and pumpkin sit inside the label where a name, subject, or date would normally go. The design does not ask you to read a slogan. It asks you to recognize a format.

I have written before about the difference between a slogan and a usable promise. A slogan is a phrase that sounds like a commitment but does not specify a trigger, an owner, or a failure mode. A usable promise is narrower and more honest: it says what will happen, under what conditions, and what it will cost. The composition notebook cover is closer to the second category, even though it contains no promise at all. It is a format with rules. You know what the label is for. You know where the binding is. You know the object is closed. The ghost and pumpkin are the only variables, and they are small enough to read as a seasonal annotation rather than a claim.

That is why the design works as a group costume without needing a caption. The shirt does not say “teacher” or “student” or “writer.” It says notebook. The viewer supplies the rest. In a group setting, the repetition of the same cover across several people turns the format into a pattern, and the pattern does the work that a printed phrase would otherwise have to do. This is not a historical claim about costume design; it is an observation about how the artwork reads. The notebook cover is recognizable enough that the ghost and pumpkin can be minor details rather than the main event.

There is a practical tradeoff in wearing a design this quiet. A shirt with a large slogan announces its intent immediately. A shirt with a notebook cover asks the viewer to complete the reference. Some viewers will not. That is the cost: the design is legible to anyone who has handled a composition notebook, and nearly invisible to anyone who has not. For a teacher group costume, that tradeoff is probably acceptable, because the group supplies the context. For an individual wearing it off-duty, the shirt reads as a graphic rather than a statement. Both readings are consistent with the artwork.

What I find interesting is the label. The white ruled rectangle headed COMPOSITION is the part of the design that carries the most information, and it is also the part that is easiest to overlook. The ghost and pumpkin are inside it, which means the seasonal elements are contained by the format rather than placed on top of it. That containment is what keeps the design from becoming a generic Halloween graphic. The notebook cover is not a background for a ghost; the ghost is an entry in the notebook. The distinction is small, but it changes how the shirt reads. One version is a costume. The other is a visual joke about where the costume is recorded.

We keep a decision log because memory is unreliable under pressure. A notebook cover is a reminder that the log has a front and a back, a binding and a label, and that the closed state is still part of the object. The shirt reproduces that closed state faithfully enough to be recognized. It does not claim to improve anything, teach anything, or solve anything. It is a printed surface that borrows the authority of a familiar format. That is a modest thing for a shirt to do, and it is enough.

What we’d do differently

If we were writing the product description, we would lead with the visual observation rather than the occasion. The notebook cover is the object; the teacher group costume is one use. Leading with the occasion makes the design sound like a prop, when the more interesting fact is that the artwork treats the cover as a complete graphic. We would also avoid implying that the shirt is a notebook or that it comes with one. The design is a printed image of a closed notebook cover, and the ghost and pumpkin are inside the label. That is the whole of it.

A break-glass account is a pre-staged emergency credential. Its value is not that it logs in fast. Its value is that when it logs in, everyone knows, and the session can be traced back to a named human afterward. For a team of two to fifteen engineers who run their own on-call rotation, that distinction matters more than it does in an organization with a dedicated security operations center.

The failure mode the account exists for is specific: the identity provider is down, the MFA device is lost, the last global administrator has left, or the federation path is unreachable. Microsoft’s guidance on emergency access accounts in Entra ID names these cases directly and states the rule plainly: use emergency access accounts only for emergency or break-glass scenarios where normal administrative accounts cannot be used, and restrict their use to times when it is absolutely necessary. The same logic applies whether your control plane is Entra ID, a self-hosted LDAP directory, or a cloud provider’s IAM.

Design constraints for a small team

Microsoft recommends creating two or more emergency access accounts, as cloud-only accounts on the *.onmicrosoft.com domain that are not federated or synchronized from on-premises. The redundancy is not decorative. A single emergency account is a single point of failure that happens to be the thing you reach for when everything else has already failed.

The federation independence is the load-bearing part. If your emergency account authenticates through the same identity provider that just went down, you have not built an emergency path. You have built a second door with the same broken lock. For a bare-metal or self-hosted environment, the equivalent is a local account that does not depend on the central directory, with credentials stored somewhere that does not require the network to reach.

Microsoft also specifies that emergency access accounts should be excluded from Conditional Access policies that block or restrict sign-in, because an enforced policy could prevent sign-in during the exact emergency the account is designed for. Report-only policies do not block access and do not need an exclusion. This is a tradeoff worth stating plainly: you are deliberately carving a hole in your normal access policy. The hole is the control. It is not a bug to be closed later.

The authentication method for the emergency account should differ from the methods used by normal administrative accounts. If your normal admin accounts use an authenticator app, the emergency account should use a FIDO2 security key or certificate-based authentication. The credential or device must not expire or fall under automated cleanup for lack of use. An emergency account that gets swept by a dormant-account policy is an emergency account that will not work when you need it.

Make the account conspicuous by design

Yale’s HIPAA break-glass procedure, which draws on a 2004 whitepaper from the Joint NEMA/COCIR/JIRA Security and Privacy Committee, describes the design principle well: emergency accounts should be created in advance so that careful thought can go into the access controls and audit trails associated with them, and the username should be obvious and meaningful, such as breakglass01, so that the account name would be inappropriate under normal operations and would stand out in audit trails.

That is the opposite of how most credentials are named. Normal accounts blend in. The emergency account should not. When someone greps the audit log six months from now, the account name should be the first thing that catches their eye.

The distribution method should also be physically evident. Yale’s procedure lists several options: kept behind glass in a cabinet where access requires literally breaking the glass, maintained within sealed envelopes where a broken seal is an obvious indication of access, locked in a desk drawer that only specific people can access, or locked in a safe or cabinet where one person knows the combination and a different person has the key to the room. The common thread is that access to the credential leaves a physical trace.

For a small team, the two-person pattern is practical. One engineer holds the safe combination; another holds the office key. Neither can reach the credential alone. The friction is the point. It is not a bug to be optimized away.

The distribution procedure should include a sign-out method requiring an acceptable form of identification, recorded before the accounts are made available. Yale’s procedure notes that this assures activities performed using the emergency account may eventually be associated with an authorized individual, creates accountability, and can assure non-repudiation. In practice, that means a paper log or a shared document with a timestamp, a name, and a reason. The reason field is not bureaucracy. It is the first line of the postmortem.

Wire the alert before you need it

Microsoft’s guidance is direct: monitor sign-in and audit log activity from the emergency accounts and trigger notifications to other administrators, so that you can verify these accounts are only used for testing or actual emergencies. The alert is the account’s real security control. If a break-glass sign-in does not page a human within minutes, the audit trail is archaeology rather than detection, and the account is effectively a permanent backdoor with a nice name.

For a team without a SOC, the alert should route to the same on-call channel that handles production incidents. The alert should fire on every sign-in, not on a threshold. There is no legitimate reason for a break-glass account to sign in without a human knowing about it in real time.

The audit record itself should contain enough to reconstruct the event. NIST SP 800-53 Rev. 5, as reflected in the CSF reference, specifies that audit records should establish what type of event occurred, when it occurred, where it occurred, the source of the event, the outcome, and the identity of any individuals, subjects, or objects associated with the event. For a break-glass session, that means the account name, the source IP or workstation, the timestamp in UTC, the actions taken, and the outcome. If your system cannot produce that, Yale’s procedure is explicit: break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead.

NIST SP 800-92 frames log management as the process for generating, transmitting, storing, analyzing, and disposing of computer security log data, and notes that it is essential to ensuring that computer security records are stored in sufficient detail for an appropriate period of time. For a small team, the practical version is: ship the break-glass account’s logs to a destination that the account itself cannot modify, and keep them long enough to cover your incident review cycle.

Write the post-use runbook

After any use of an emergency access account, Microsoft’s guidance calls for a review to determine whether the use was authorized and whether the actions taken were appropriate. That review has a few concrete steps.

First, preserve the logs. Capture the sign-in record, the audit trail of privileged actions, and any session recordings or command history the platform provides. NIST’s audit guidance specifies that audit record reduction and report generation should support on-demand review and after-the-fact investigations without altering the original content or time ordering of audit records. Copy the raw records before you summarize them.

Second, reconcile the session to a named human. The sign-out log from the distribution procedure gives you the name. The audit trail gives you the actions. The postmortem gives you the reason. If those three do not line up, that is the finding.

Third, disable or rotate the credential. Yale’s cleanup procedure recommends disabling or deleting the emergency account that was used to prevent reuse now that the password is known, and reconciling the data acquired and audit trails to reflect the proper operator’s name. Some systems can automatically deactivate emergency accounts after first use or after a selectable period such as 8 hours or 1 day. Commvault’s break-glass implementation takes a different approach: the password is regenerated on every login to prevent repeated use of the same password, and all actions are recorded in the audit trail. Either pattern works. What does not work is leaving a known password in place after an incident that involved a dozen people in a channel.

Fourth, ask whether the normal access path failed in a way that should be fixed. Yale’s procedure notes that frequent use of emergency accounts may indicate problems with the normal user authentication mechanism. A break-glass account that gets used every month is not a break-glass account. It is a helpdesk with extra steps, and Yale’s procedure explicitly says break-glass should not replace a helpdesk.

Schedule the drill

Microsoft recommends performing account validation drills at least every 90 days, including reviewing the authorized-user list and testing sign-in and administrative task functionality. The 90-day cadence is a reasonable default for a small team. The important part is the second half of that sentence: testing administrative task functionality, not just sign-in.

A drill that only tests sign-in is not a drill. The failure that matters is discovering at 3 a.m. that the credential works but the permission it grants does not cover the task. If the break-glass account is supposed to let you restore a database, the drill should include a restore. If it is supposed to let you rotate an IAM role, the drill should include a role switch. AWS IAM documentation notes that when you switch to a role in the console, you temporarily give up your user permissions and work with the permissions assigned to the role, and when you exit the role, your user permissions are automatically restored. That is a useful property for a break-glass path, but only if the role’s permissions actually cover the emergency task.

One more AWS-specific detail worth knowing: you cannot switch roles in the AWS Management Console to a role that requires an ExternalId value. Such a role can only be assumed by calling the AssumeRole API that supports the ExternalId parameter. If your break-glass path depends on a role with an ExternalId, the console switch will not work, and the drill will find that out before the incident does.

Treat a failed drill as an incident. It is a control that did not work when tested under controlled conditions. The blameless postmortem format applies: what did we expect, what happened, what will we change.

The tradeoff to accept

The break-glass account is slower and more awkward than a normal login on purpose. Two people, a physical envelope, a sign-out log, a page to the on-call channel, a postmortem the next morning. Every one of those steps is friction, and every one of those steps is the control.

The alternative is an account that is fast and quiet. That account is not a break-glass account. It is a standing privilege with a reassuring name, and the first time it is used without anyone noticing, you will have lost the ability to tell an emergency from a misuse.

If you are building the recovery checklist that the break-glass account supports, the Recovery Checklist Before You Need It is a useful companion. The break-glass account gets you into the system. The checklist tells you what to do once you are there.

Questions that come up

How many break-glass accounts should a small team have? Microsoft recommends two or more for redundancy. For a team of two to fifteen, two is the practical minimum. One is a single point of failure. More than two multiplies the credential storage and rotation burden without adding much resilience.

Should the break-glass account be a person or a shared account? Microsoft’s guidance says not to associate emergency access accounts with any individual user in the organization, and to store credentials in a known secure location available to multiple members of the administration team. The account is a shared emergency path, not a personal login. The sign-out log is what ties a specific session to a specific person.

What if the system cannot produce an audit trail? Yale’s procedure is explicit: if the system cannot provide an audit trail showing simple account activity like login attempts, break-glass use needs to be carefully considered before implementing, and may require a manual paper log instead. A break-glass account without an audit trail is a standing privilege with no accountability.

How often should the credential be rotated? The sources describe per-login regeneration (Commvault) and post-use disable-or-delete (Yale), but do not specify a fixed rotation interval for offline-stored credentials. The 90-day validation drill is the cadence the sources support. Rotation after every use is the safer default; a fixed calendar rotation is a secondary control.

Does the break-glass account replace normal access controls? No. The sources frame it as a last resort alongside standard controls. Yale’s procedure says standard access controls should be established with sufficient rules to minimize the number of times break-glass needs to occur. The account is the exception path, not the primary one.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The garment preview shows one word: meh. Lowercase. Bold. White on a dark shirt. No animal. No subtext. No parenthetical. The period is the only punctuation, and it lands like a closed door.

I keep looking at it because it refuses to do the thing most slogan shirts do. It doesn’t ask a question. It doesn’t list a hobby. It doesn’t wink at a fandom. It states a verdict and stops. For a team that spends its days writing runbooks, escalation policies, and decision records, that restraint is worth studying.

The shirt is listed as Meh Funny Sarcastic Saying Introvert Dry Humor under the Quirky Cotton Collective brand. The retail title stacks three descriptors — funny, sarcastic, introvert, dry humor — but the artwork itself carries none of that scaffolding. The design is the word. Everything else is metadata.

What the artwork actually does

Large bold lowercase typography on a dark ground is a visual choice with consequences. Lowercase reads as casual, unbothered, almost mumbled. Bold weight makes it loud. White on dark gives it contrast. The combination is a contradiction: a shout that says nothing is worth shouting about.

That contradiction is the joke, if there is one. The word meh is a shrug. Rendering it in the largest available type is a way of shrugging theatrically. The shirt doesn’t claim the wearer is indifferent to everything; it claims the wearer has decided this particular moment — the moment of reading the shirt — does not merit enthusiasm.

I’m not going to pretend this is a deep statement about labor, burnout, or the human condition. It’s a word on a shirt. But the word is doing something specific: it is refusing to promise anything. And that refusal is the most interesting thing about it.

Slogans versus promises

In operations work, we spend a lot of time distinguishing between a slogan and a usable promise. A slogan is a phrase that sounds good in a meeting. A usable promise is a sentence you can test. “We prioritize reliability” is a slogan. “We page a human within five minutes of a failed synthetic check” is a usable promise. One of them survives an incident; the other becomes a postmortem bullet point.

The meh shirt is not a usable promise. It is not trying to be. It is a slogan that knows it’s a slogan. It doesn’t claim the wearer is calm under pressure, or that they have a five-nines uptime record, or that they never lose their temper in a code review. It claims they are unimpressed. That’s a mood, not a metric.

I find that honesty refreshing. Most slogan shirts overpromise. They imply a personality trait — “I’m the funny one,” “I’m the smart one,” “I’m the one who fixes your computer” — and then the wearer has to live up to it. The meh shirt implies nothing. It is a preemptive withdrawal from the conversation. You can’t be disappointed by someone who never claimed to be impressive.

The introvert framing

The retail title includes “Introvert,” and the related search queries include “meh introvert shirt.” I understand why. The word meh functions as a boundary marker. It says: I am here, I am participating at the minimum required level, and I would like to go home now. For people who find social interaction draining, that is a recognizable posture.

But the artwork itself does not say “introvert.” It says meh. The introvert reading is an interpretation, not a printed claim. I think that’s important. The shirt is not a diagnosis. It is a mood that happens to align with a certain temperament. You can wear it as an introvert, an extrovert, or someone who just doesn’t want to talk about the weather. The word doesn’t check your credentials.

In team settings, this matters. A shirt that says “I’m an introvert” invites a conversation about personality types. A shirt that says meh invites nothing. It closes the loop. For someone who wants to signal a boundary without explaining it, that’s efficient.

What it doesn’t do

I want to be clear about what this shirt is not. It is not a diagnostic tool. It does not measure burnout, introversion, or operational fatigue. It does not cure anything. It does not improve team communication. It is a piece of clothing with a word on it.

I also can’t tell you how it fits, how it wears, or how it holds up after a wash. I haven’t worn it. I’m looking at the artwork as observed in the native garment preview: large bold white lowercase meh. lettering on a dark shirt, no animal illustration present. That’s the object. Everything else is inference.

What I can say is that the design is legible. The word is the whole design. There is no competing element, no decorative flourish, no hidden meaning. It is a single word, rendered large, and it does not pretend to be more than that.

Why this matters for technical readers

If you work on a small team, you are probably familiar with the gap between what a system promises and what it delivers. You write a runbook that says “restore from backup,” and then you discover the backup is corrupt. You write an escalation policy that says “page the on-call,” and then you discover the on-call is asleep. The gap between slogan and promise is where incidents live.

The meh shirt is a reminder that not everything needs to be a promise. Some things are just statements. Some things are just moods. Some things are just a word on a shirt. And that’s fine. Not every object has to carry the weight of a reliability practice.

But I do think there’s a small lesson here about honesty in language. The shirt doesn’t overclaim. It doesn’t say “I’m fine.” It doesn’t say “I’m thriving.” It says meh. That’s a low bar, and it clears it. In a world of overpromising slogans, there’s something to be said for a garment that promises nothing and delivers exactly that.

What we’d do differently

If we were designing a shirt to communicate a boundary, we would probably not choose a single word. We would choose a sentence with a subject and a verb, something testable. But that would be a different object. The meh shirt is not a boundary statement; it is a mood statement. And moods don’t need to be testable.

What we would do differently is resist the urge to over-interpret it. It is a word on a shirt. It is not a manifesto. It is not a diagnosis. It is not a promise. It is a shrug, rendered in bold lowercase type, and it is content to be exactly that.

Sometimes the most honest thing you can print is the thing you don’t bother to capitalize.