Editorial note: This publication and Quirky Cotton Collective share an owner.

The shirt shows a white curved ramp descending from left to right. Above it, in white capitals: I HAD POTENTIAL. At the upper left ledge sits a dashed empty circle. At the lower right, on the flat runout, rests a solid white ball. The ramp is curved, not a straight diagonal. That curve is the whole argument.

We keep coming back to this design because it is not a slogan with a picture attached. It is a diagram with a sentence attached, and the two are in tension. The sentence is past tense and first person: I had potential. The drawing is present tense and third person: a ball, a ramp, a dashed outline where something used to be. The joke only works if you read both. The dashed circle is the potential — an outline, a position, a shape that could have been occupied. The solid ball is what actually happened. It rolled down the curve and came to rest on the flat. Nothing exploded. Nothing was destroyed. The ball is fine. It is just no longer at the top.

That is a more precise joke than the phrase alone suggests. A straight diagonal would have made the same point faster and worse. A straight ramp implies a single slope, a clean acceleration, a predictable outcome. The curve implies something else: the path changed as it went. The ball did not fall off a cliff. It followed a shape. The shape was there before the ball arrived. That is the part we find useful as an observation, not as a lesson.

We spend a lot of working hours around diagrams that describe systems rather than people. A runbook is a diagram of a procedure. A decision tree is a diagram of a choice. An escalation policy is a diagram of who gets woken up. The best of them share a property with this shirt: they show the shape of the path, not just the endpoint. A runbook that says “restore the database” is a straight diagonal. A runbook that says “check the WAL archive gap before you promote the replica, because the last drill showed a 40-second window where the primary and standby disagreed” is a curve. It costs more to write. It is more useful at 3 a.m.

The shirt is not a runbook. We are not going to pretend it demonstrates anything about backup recovery, incident response, or the reliability of a small team’s on-call rotation. It is a piece of printed cotton with a physics-adjacent visual pun on it. But the visual pun is doing real work, and it is worth naming what that work is. The dashed circle is a counterfactual. The solid ball is an observation. The gap between them is the joke. Most slogan shirts collapse that gap by asserting the punchline outright. This one leaves the gap visible and lets the reader close it.

There is a second reading, and we think it is the one that makes the shirt worth wearing rather than just worth noticing. The ball is not lost. It is resting. The ramp did not fail; it delivered the ball to a flat ledge, which is where balls go when ramps end. The sentence says had potential, but the drawing says reached equilibrium. Those are not the same claim. One is a regret. The other is a description of a system that has settled. We are not going to argue that the shirt is secretly optimistic — that would be overreading a garment — but we will note that the artwork is more ambiguous than the text. The text is a verdict. The drawing is a state. If you have ever written a postmortem that concluded “the system behaved as designed, and the design was the problem,” you have lived in that gap.

The design is also unusually restrained for a novelty shirt. White ink on a dark ground, no secondary colors, no distressed texture, no additional copy. The only punctuation is the period implied by the end of the sentence, which is not printed but felt. The dashed circle is the lightest element on the shirt and the most important. It is the only part of the drawing that is not solid, and it is the only part that refers to something that is not there. That is a deliberate choice, and it is the kind of choice we notice because we make similar ones in diagrams: the empty box, the dotted line, the placeholder that says this is where the thing would go if it existed. A dashed outline is how a diagram admits a counterfactual without asserting it.

We keep a small decision log for our own work. One of the entries we return to is a note about the difference between a promise and a description. A promise says what will happen. A description says what did happen. Most operational writing fails by confusing the two: the runbook promises a clean failover, the postmortem describes a 90-second window of split-brain, and the gap between them is where the next incident lives. The shirt is a description. It describes a ball on a ledge and a dashed circle on a ramp. It does not promise that the ball will roll back up. It does not promise that the dashed circle will be filled. It just shows you the shape and lets you decide what to do with it.

If you are the kind of reader who keeps a pager and a decision log, the shirt is available as a standard t-shirt from Quirky Cotton Collective. We are not going to tell you it will improve your on-call rotation. It is a shirt. But it is a shirt with a diagram on it, and the diagram is more careful than it needed to be. That is worth something.

What we’d do differently

If we were writing the copy for this shirt, we would resist the urge to explain the joke in the product title. The current title stacks “funny,” “physics pun,” “sarcastic,” and “minimalist” into a single line, which is four claims where the artwork makes one. The artwork is minimal. The title is not. We would let the diagram do the work and keep the retail prose closer to the observation: a curved ramp, a dashed circle, a solid ball. That is the whole object. Everything else is a search keyword wearing a hat.

We would also keep a note in our own decision log about the difference between a dashed outline and a solid one. The dashed circle is the part of the design that carries the meaning, and it is the part most likely to be lost in a thumbnail. If we were designing a diagram for a runbook, we would ask whether the empty box is doing the same work: is it showing a counterfactual, or is it just a placeholder we forgot to fill? The shirt is a reminder that the empty shape can be the point. We would write that down before the next incident, not after.

Two clocks, one row

Most postmortem timelines have a single row that reads like this: 14:07 — alert fired; on-call investigated and mitigated. That row hides two different failures. Detection asks how long it took for any signal to reach a human. Diagnosis asks how long it took for a human to hold a correct model of what was breaking. A lean team that collapses both into one timestamp cannot tell whether its next investment belongs in monitoring or in runbooks, access, and observability.

This is a writing and process problem, not a tooling problem. The templates most teams already use do not have a dedicated diagnosis field. PagerDuty’s published postmortem template, for example, asks for the time the contributing factor began, the time of the page, the time the status page was updated, the time of significant actions, and the end time, each with a link to the tool or log that shows how the timestamp was arrived at. It does not ask when someone first understood the failure. The fix is to add boundary rows to the template you have, with the same evidence discipline the template already demands.

Define four boundaries before the incident

The SRE Book is direct that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary. The same logic applies inside the document. Agree on four boundary events in advance, and write them into the template as empty rows:

1. Failure onset. When the system first departed from normal behavior, whether or not anyone could see it. This is often reconstructed from logs and is the least certain row in the table.
2. First signal. When any signal reached a human or a system that would act on it: an alert firing, a synthetic check failing, a customer report arriving. This is detection time.
3. First correct diagnosis stated aloud. When someone wrote or said a hypothesis that turned out to be the actual cause. This is diagnosis time. It is not the same as acknowledgment, and it is not the same as the first hypothesis.
4. First mitigation action. When someone changed the system to reduce impact, even if the change was temporary.

Two derived gaps fall out of these rows: signal-to-human (detection) and human-to-correct-model (diagnosis). They point at different remediation work. A long detection gap is a monitoring or coverage problem. A long diagnosis gap is a context, runbook, access, or observability problem. An action item that does not name which gap it closes is hard to prioritize six weeks later.

Use the existing template as a floor, not a ceiling

PagerDuty’s template is a good floor because it insists on traceability: every timestamp should link to the tool or log that justifies it. Keep that discipline and extend the table. A workable column set for a small team is:

The caveat column matters more than it looks. Monitoring data has its own lag, and a timeline that treats a dashboard timestamp as ground truth can understate or overstate detection. PostgreSQL’s documentation notes that each server process flushes accumulated statistics to shared memory just before going idle, but not more frequently than once per PGSTAT_MIN_INTERVAL milliseconds (one second unless altered at build time), so displayed totals lag actual activity. The same documentation notes that in the default configuration, statistics accessed within a transaction are cached until that transaction ends, so a long-running query can show static numbers. On the AWS side, CloudWatch preserves alarm history for 30 days and marks each state transition with a unique timestamp, and an alarm invokes actions only when it changes state. If your first signal came from an alarm that had been in ALARM for twenty minutes before anyone looked, the alarm history will show the transition, not the human noticing. Record both.

Three rows, not one: paged, acknowledged, understood

For a team of two to fifteen engineers who are their own on-call rotation, the most common diagnosis gap is not ignorance. It is context switching. The on-call is mid-task, acknowledges the page, finishes a sentence or a deploy step, and only then begins real diagnosis. If the timeline records only “acknowledged,” that gap is invisible and the postmortem will misattribute the delay to tooling.

Split the human side into three rows:

• Page fired. The alert or escalation event, with its tool timestamp.
• Acknowledged. The moment a human claimed the page. This is a tool log, not a memory.
• First correct diagnosis. The moment a correct hypothesis was stated in chat or on a call. This is usually reconstructed, and should be labeled as such.

The middle row is the one lean teams most often skip, and it is the one that makes the context-switch cost visible without blaming the person. Write it as a system state: “page acknowledged while a deploy was in progress,” not “on-call was slow to respond.”

Log the hypotheses that died

Diagnosis is iterative, and the final answer is rarely the first guess. A timeline that records only the correct diagnosis reads as if the team knew immediately, which makes the diagnosis gap look shorter than it was and hides the evidence that killed the wrong hypotheses. Add a short hypothesis log beneath the boundary table:

H1 (14:12): Primary overloaded by traffic spike. Killed at 14:18 — request rate flat in the load balancer metrics.
H2 (14:19): Network partition between primary and replica. Killed at 14:23 — replication connection alive, lag growing steadily.
H3 (14:26): WAL retention exhausted, replica cannot catch up. Confirmed by archive status and disk usage.

Each hypothesis carries the evidence that killed it. This is the decision-log part of the timeline, and it is what a reviewer needs to re-derive the diagnosis gap rather than trust the author’s summary. It also produces better action items: H1 and H2 were killed by metrics that existed, which suggests the diagnosis gap was not a missing-signal problem. H3 required a check that lived in a runbook nobody had opened, which suggests a different fix.

Keep the language factual

Blamelessness is a formatting constraint as much as a cultural one. The SRE Workbook’s critique of a poorly written postmortem calls out animated language and dramatic descriptions as distractions that erode psychological safety, and recommends providing verifiable data to justify the severity of a statement. The SRE Book defines a blameless postmortem as one that focuses on contributing causes without indicting any individual or team, and assumes everyone involved did the right thing with the information they had. Google’s CRE guidance goes further for shared documents: express the role of humans in terms of role rather than name.

In practice, that means the timeline says “the on-call” and “the alert,” not names. It says “the runbook did not cover WAL retention” rather than “the on-call did not know.” It says “the alert fired at 14:07 and was acknowledged at 14:11” rather than “response was slow.” The boundary rows make the gaps visible; the language keeps them reviewable.

A worked example, clearly hypothetical

Consider a PostgreSQL replica-lag incident on a small team running pgBackRest and streaming replication. The alert fires at T+0. The on-call acknowledges at T+4 while finishing a deploy. The first hypothesis is a traffic spike; it dies at T+11 when the load balancer metrics are flat. The second hypothesis is a network partition; it dies at T+17 when the replication connection is shown alive. The correct diagnosis — WAL retention exhausted on the primary, so the replica cannot catch up — lands at T+19. Mitigation begins at T+24 when a batch writer is paused.

The boundary table looks like this:

• Detection gap: T+0 (signal) minus failure onset. If onset was T−6, detection took six minutes.
• Acknowledgment gap: T+4 minus T+0. Four minutes of context switching.
• Diagnosis gap: T+19 minus T+4. Fifteen minutes of sense-making, including two dead hypotheses.
• Mitigation gap: T+24 minus T+19. Five minutes to act once the model was correct.

Now attach action items to rows. The detection gap suggests a review of the alert’s evaluation window and whether a leading indicator existed earlier. The acknowledgment gap suggests either an escalation policy change or an honest acknowledgment that a single on-call rotation will always have context-switch cost, which is a staffing tradeoff rather than a tooling bug. The diagnosis gap suggests a runbook entry for WAL retention and a dashboard panel that shows archive status next to replica lag. The mitigation gap suggests a pre-approved action — pausing the batch writer — that does not require a second pair of eyes at 2 a.m.

None of these action items is novel. What is novel is that each one names the row it closes, which makes the postmortem review meeting a prioritization exercise rather than a re-litigation of the incident.

What the timeline is for

A postmortem timeline with explicit boundaries is a decision log, not a narrative. The SRE Workbook notes that a good postmortem lets readers know what happened, how the issue was mitigated, and how users were impacted — and that the recovery section is where many drafts fall short. The boundary rows are how the recovery section gets specific. The SRE Book lists a monitoring failure, which usually implies manual incident discovery, among common postmortem triggers; a timeline that separates detection from diagnosis is how you tell whether a given incident was that kind of failure or a different one.

For a team without a dedicated reliability function, the timeline is also the only durable record of how the on-call rotation actually behaves under load. The recovery checklist you write before you need it — the one that covers restores, access, and escalation — is a different artifact, but it shares the same discipline: name the step, name the evidence, name the owner. If your team is still building that habit, the recovery checklist article is a useful companion to this one.

Review checklist

Before the postmortem meeting, check the timeline against these questions:

• Are all four boundaries present — failure onset, first signal, first correct diagnosis, first mitigation?
• Is each timestamp linked to a log, alert history entry, or chat transcript, with a note on any known lag or retention limit?
• Are “page fired,” “acknowledged,” and “first correct diagnosis” separate rows?
• Are the dead hypotheses recorded with the evidence that killed them?
• Does each action item name the gap it closes?
• Is the language factual — roles rather than names, system states rather than judgments?

If the answer to the first three is yes, the timeline will survive review. If the answer to the last three is yes, the action items will survive the next incident.

FAQ

What if we genuinely cannot reconstruct the diagnosis time?

Say so. Label the row “reconstructed from chat, approximate” and give the range. A labeled estimate is more useful than a false precision, and the SRE Workbook’s guidance to provide verifiable data applies to the caveat as much as to the timestamp.

Does this add process overhead during the incident?

No. The boundary rows are filled in after the incident, from logs and chat. The only in-incident change is that someone notes the first correct hypothesis when it is stated, which is usually already happening in the incident channel. If your team uses a scribe role, that is the scribe’s job.

Should we add these fields to our template permanently?

Yes, as empty rows. A template that asks for the diagnosis boundary will get one; a template that does not will get a narrative. The PagerDuty template’s insistence on evidence links is the model to copy.

What about incidents where diagnosis and mitigation overlap?

Record both rows anyway. Overlap is information: it usually means the mitigation was a diagnostic action, such as restarting a process to see whether the symptom clears. That pattern is worth naming in the postmortem because it often indicates a missing observability signal.

A maintenance window is a planned change with a known failure surface. That makes it a good candidate for a pre-mortem: a short exercise in which the team assumes the change has already failed and writes down why. Gary Klein’s description of the technique is blunt about the mechanism — people are reluctant to voice reservations during planning, and a pre-mortem makes it safer for knowledgeable dissenters to speak up (Harvard Business Review, 2007).

For a team of two to fifteen engineers who are their own on-call rotation, the pre-mortem is not a risk register and not a prediction. It is a drill-design tool. Its output should be a set of rehearsable checks with pass/fail results, each tied to one named failure mode. The window gets scheduled when the checks have results, not when the team has promised to be careful.

Frame the exercise before you frame the migration

Klein’s instruction is to assume the project has already failed and then generate plausible reasons. Applied to a PostgreSQL migration, the opening sentence is not “what could go wrong” but “the window is over and it went badly — what happened?” The answers tend to cluster around a small number of surfaces: the backup, the archive, version and extension compatibility, the rollback path, and the people.

Run the session with the people who will actually be on the call. In a small team that is usually everyone who touches the database, plus whoever owns the application that reads from it. One person writes; the writer is not the decision-maker. If the team has no dedicated facilitator, rotate the writing role and keep the decision-maker’s hands off the keyboard so they can listen.

The pre-mortem should be run against the rollback plan as hard as against the forward plan. If the team cannot state in one sentence what “abort” means and who says it, the exercise has found its most important gap before it has found anything else.

Failure mode 1: the backup you would restore from is not the backup you think it is

The first plausible reason the window failed is that the restore did not work. This is the failure mode most worth converting into a drill, because it is cheap to test and expensive to discover at 02:00.

PostgreSQL’s documentation is explicit that recovery from a continuous archive requires a continuous sequence of archived WAL files extending back at least as far as the start time of the backup (PostgreSQL 18, Section 25.3). It also advises setting up and testing the WAL archiving procedure before taking the first base backup. The pre-mortem check that follows is concrete: restore a base backup plus the WAL needed to reach a chosen recovery target into a scratch instance, on hardware that is not the production host, before the window. Record the recovery target, the command used, and the time to completion.

If the team uses pgBackRest, the documentation states that a restore requires the backup files and one or more WAL segments to work correctly (pgBackRest User Guide). That is the minimum the drill must exercise. A restore that only copies files and never starts the cluster has not tested the thing the pre-mortem is worried about.

For file-level backups outside PostgreSQL — configuration directories, application secrets, the runbook repository — the same logic applies. restic’s documentation describes repository integrity checking and snapshot listing, but integrity checking is not the same as a restore drill. The pre-mortem should name which restic snapshots the migration depends on and who will restore one to a scratch path before the window.

Failure mode 2: WAL archiving silently stopped

The second plausible reason the window failed is that the archive had a gap the team did not notice. PostgreSQL’s documentation is precise about the contract: the archive command must return zero exit status if and only if it succeeds; a nonzero status tells PostgreSQL the file was not archived and it will retry periodically (Section 25.3.1). The same section warns that if archiving falls significantly behind, more data would be lost in a disaster, and pg_wal/ can accumulate not-yet-archived segments until disk space is exhausted.

The pre-mortem check is a gate, not a glance. Before the window, confirm that the most recent archived segment is recent, that the archive command’s exit behavior is what the team believes it is, and that the monitoring that would catch a stalled archiver is actually firing. If the team uses pgBackRest, note that the documentation requires local and remote pgBackRest versions to match exactly, and that a mismatch prevents WAL archiving and backups from functioning until the versions match (pgBackRest User Guide). A version skew introduced during migration preparation is exactly the kind of quiet failure the pre-mortem exists to surface.

One more check that belongs here: WAL archiving does not restore changes to postgresql.conf, pg_hba.conf, or pg_ident.conf, because those are edited manually rather than through SQL (Section 25.3). If the migration changes any of those files, the pre-mortem should name where the new versions are stored and how they get onto the restored host.

Failure mode 3: version and extension incompatibility

The third plausible reason the window failed is that the target version would not accept something the source version allowed. PostgreSQL’s documentation notes that for major releases the internal data storage format is subject to change, that dump-and-restore is the traditional method, that pg_upgrade is faster, and that replication methods are also available (Section 18.6). The same section recommends that cautious users test client applications on the new major version before switching over fully, and suggests concurrent installations of old and new versions.

Extension compatibility is where this failure mode usually bites. Azure Database for PostgreSQL flexible server documents upgrade validation checks that evaluate compatibility and configuration without initiating the upgrade, and states that these checks do not modify the server version, trigger downtime, or restart the server (Microsoft Learn). The same documentation states that in-place major version upgrades do not support all PostgreSQL extensions, and that the upgrade fails during precheck if a blocked extension is present on an affected upgrade path. That is a useful model even for teams not on Azure: enumerate every extension, every version-specific blocker, and every object that depends on an extension, and resolve them before the window rather than during it.

The pre-mortem check is an inventory with a result per row: extension name, installed version, target-version support, owner, and disposition. A row without a disposition is a check that has not passed.

Failure mode 4: no tested rollback

The fourth plausible reason the window failed is that the team could not get back. Azure’s documentation is direct about the shape of this problem: after a successful in-place major version upgrade there are no automated ways to revert to the earlier version, though point-in-time recovery to a time before the upgrade can restore the previous version on a new server (Microsoft Learn). Self-managed PostgreSQL has the same property in different clothing: the rollback is a restore, and a restore is a procedure that has to be rehearsed.

The pre-mortem should force the team to write the abort criterion in one sentence and name the person who calls it. It should also force a second sentence: what the rollback actually is — a restore to a new host, a DNS switch back to the old primary, a logical replication reversal — and how long it takes. If the answer is “we would figure it out,” the pre-mortem has found a gap that no amount of scheduling discipline will close.

Failure mode 5: the on-call rotation is the same two people

The fifth plausible reason the window failed is staffing. In a team that is its own on-call rotation, the migration window competes with whatever else pages that night. The pre-mortem should include a staffing failure mode: who is awake, who is the decision-maker, who is the scribe, and what happens if the window overruns into the next working day.

This is a planning argument, not a claim about any specific team. The point is that the pre-mortem is the right place to notice that the person who knows the extension inventory is also the person who would be paged for an unrelated incident at the same hour. Assign roles in advance: operator, scribe, decision-maker. Write a stop-work condition — a time, a metric, or an event — that ends the window regardless of progress.

Convert each failure mode into a check with an owner and a result

The pre-mortem is only useful if it changes what happens before the window. The conversion is mechanical: every plausible failure becomes a check, every check gets an owner and a deadline, and every check returns a pass, a fail, or a not-run. The window is scheduled when every check has a result. A promise to be careful is not a result.

A hypothetical example, clearly labeled as such: a two-person team plans a PostgreSQL major-version window. The pre-mortem lists “the extension we forgot is blocked on the target version” and “the archive command has been failing quietly for a week.” Each becomes a check with an owner and a deadline before the window. The first check is an extension inventory reconciled against the target version’s release notes. The second is a review of the archiver’s recent history and the monitoring that would have caught a stall. The window is scheduled only when both checks return a result. If either fails, the window moves; that is the pre-mortem doing its job.

This is also where the pre-mortem connects to the recovery checklist the team should already have. Writing the recovery checklist before you need it is the same discipline applied to a different artifact: Write the Recovery Checklist Before You Need It describes the checklist as something you build and test in advance, not something you assemble during an incident. The pre-mortem is the moment to confirm that the checklist exists, that it names the current backup repository, and that someone has run it end to end.

After the window: read the postmortem against the pre-mortem

Google’s SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause or causes, and the follow-up actions to prevent recurrence (Site Reliability Engineering, Chapter 15). It also states that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary, and that a blameless postmortem focuses on contributing causes without indicting any individual or team.

The pre-mortem gives the postmortem a comparison set. After the window, the team can ask which predicted failures happened, which did not, and which surprises were absent from the list. Predicted failures that did not occur are as informative as unpredicted ones, because they show where the team’s model of the system is miscalibrated. A pre-mortem that predicted an archive gap and found none is not wasted; it is evidence that the archive check worked, and it should be recorded as such.

The postmortem should also record whether the abort criterion was used, and if not, why. A window that overran because no one was willing to call the abort is a finding about the plan, not about the people.

What the pre-mortem does not do

A pre-mortem reduces surprise. It does not eliminate downtime, and it is not a substitute for a tested restore. It can schedule the restore drill and name its owner, but the drill itself is the evidence. The pre-mortem only ensures the drill happens before the window rather than during it.

It also does not replace the platform’s own validation. If the team runs on a managed service that offers pre-upgrade validation checks, those checks are the authoritative source for that platform’s blockers, and the pre-mortem should consume their output rather than duplicate it. If the team runs self-managed PostgreSQL, the equivalent is the extension inventory, the version-specific release notes, and a restore drill against the actual backup repository.

The useful claim is narrow: a pre-mortem converts a list of worries into a list of checks, and a list of checks with results is a better basis for scheduling a maintenance window than a list of worries. That is enough.

FAQ

How long should a pre-mortem for a database migration take?

Long enough to enumerate the failure modes and assign owners, short enough that the people who will be on the call actually attend. For a small team, the session is usually one meeting plus asynchronous follow-up on the checks. The checks themselves take longer than the meeting; the meeting’s job is to produce them.

Who should facilitate if the team has no dedicated SRE?

Rotate the writing role and keep the decision-maker out of the scribe seat. The facilitator’s job is to keep the exercise in the past tense — “the window failed, why” — rather than letting it drift into a forward-looking risk review. If the team is two people, one writes and one talks, and they swap for the next migration.

Does a pre-mortem replace a restore drill?

No. The pre-mortem can schedule the drill and name its owner, but the drill is the evidence. PostgreSQL’s documentation advises testing the WAL archiving procedure before taking the first base backup; the pre-mortem is the mechanism that ensures the test happens before the window rather than during it.

What if the pre-mortem finds a failure mode the team cannot fix before the window?

Then the window moves, or the scope changes, or the team accepts the risk explicitly and records the acceptance. The pre-mortem’s value is that the acceptance is a decision rather than an oversight. A check that returns “fail, accepted by [name] on [date]” is a result; a check that returns nothing is not.

How does this differ from a standard change-management review?

A change review asks whether the change is approved. A pre-mortem asks what would make it fail and whether the team has rehearsed the answer. The two can share a meeting, but the pre-mortem’s output is a set of checks with results, not an approval.

The script is the least valuable thing the author leaves behind. Anyone can read a crontab line or a schedule: field. What the next on-call engineer cannot reconstruct from the script is what normal looks like, which failures are silent, what a partial run corrupts, and what to check first when the output is empty. That knowledge lives in the author’s head, and it is the actual handoff.

This is a procedure for extracting it. It treats the departing author’s cron job as a failure-mode inventory rather than a code review, and it borrows the structure of postmortem and runbook craft that already exists for larger teams. The scale is different — two to fifteen engineers, no dedicated SRE, the same people on call who write the code — but the documentation problem is the same one Google’s SRE book describes: an incident record exists so that the next person does not rediscover the failure from scratch.

Start with the failure inventory, not the rewrite

The instinct when an owner leaves is to schedule a rewrite. Resist it for the handoff window. A rewrite is a project; a failure inventory is an afternoon, and it is what makes the job survivable in the meantime.

List every way the job can fail, and for each one write what it looks like from outside the process. A working inventory for a scheduled job usually covers:

• Silent no-op. The process exits zero but wrote nothing, because an upstream query returned no rows, a filter matched nothing, or a feature flag disabled the work.
• Partial write. The job processed some records and died, leaving a half-updated table or a partially uploaded object.
• Overlapping run. A slow run is still going when the next schedule fires, and two processes write the same rows.
• Missed schedule. The scheduler did not fire, or fired late, and the job’s assumptions about the previous run’s state no longer hold.
• Credential expiry. A token, key, or certificate rotated or expired and the job failed at authentication.
• Upstream dependency down. The database, API, bucket, or queue the job reads from was unavailable or slow.

For each entry, the author should write three things: how you would notice it, what it damages if left alone, and the first check a responder should run. If the author cannot answer “how would you notice it,” that is itself the finding — the failure mode is currently invisible, and the handoff should say so plainly rather than imply coverage that does not exist.

Borrow the postmortem lens for failures that have not happened

Google’s SRE book defines a postmortem as “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” That structure is retrospective by convention, but nothing about it requires the incident to have happened yet. For a handoff, fill in the same sections as a pre-mortem: for each failure mode in the inventory, write the trigger, the impact, the detection signal, the recovery steps, and the action item that would prevent recurrence.

The SRE book also lists common postmortem triggers, including data loss of any kind, on-call intervention, and monitoring failure — the last of which “usually implies manual incident discovery.” Those triggers map directly onto cron job failure modes. A job that silently stops writing is a monitoring failure by definition. A job that duplicates rows is data loss in the sense that matters to the downstream consumer. Writing the trigger conditions down before the incident means the next on-call does not have to decide, at 2 a.m., whether this counts as an incident worth escalating.

Two craft notes from the SRE Workbook apply with unusual force to a handoff document. First, the workbook’s critique of a bad postmortem notes that its Recovery Efforts section was left empty — the record of what was actually done to fix things was missing. A handoff doc with a failure inventory but no recovery steps has the same defect. Second, the workbook recommends a single owner rather than several, because “an owner is a single point of contact who is responsible for the postmortem, follow-up, and completion.” A handoff document with four names on it and no review date is a document that will rot. Name one owner and set a review date.

Separate “did it run” from “did it work”

The most dangerous failure mode for a lean team is the one that pages no one. A cron job that exits zero and writes nothing produces no alert, no error log, and no obvious symptom until a downstream consumer notices stale data — sometimes days later. The handoff must include a success check that is independent of the exit code.

What that check looks like depends on the job, but the shape is consistent: assert something about the output that would be false if the job had not done its work. A row count in a target table. A timestamp in a heartbeat table. A file’s presence and modification time in a bucket. The point is that the check must be runnable by someone who did not write the job, and it must fail loudly when the job silently no-ops.

This is where the handoff document earns its keep. The author knows which query returns zero rows on a quiet day and which returns zero rows because something broke. The next person does not. Write down the difference.

Document the blast radius explicitly

The information the next on-call needs most urgently is also the information the author is most likely to omit: what the job touches. Which tables, buckets, queues, or downstream consumers does it write to? What does a partial run leave behind? What does a duplicate run corrupt?

For a job that writes to PostgreSQL, this means naming the tables and the write pattern — insert, upsert, delete-then-insert — because the recovery procedure differs for each. For a job that writes to object storage, it means naming the bucket and prefix and whether a re-run overwrites or appends. For a job that publishes to a queue, it means naming the consumer and whether duplicate messages are idempotent on the receiving end.

The blast radius section should also state what the job does not touch. A responder who knows the job only writes to one table will not waste time checking the others. Negative information is cheap to write and expensive to reconstruct.

Capture the schedule’s assumptions, not just the expression

A cron expression without its assumptions is a trap. The expression says when the job fires; it does not say what timezone the schedule is interpreted in, how long a normal run takes, what happens when a run is late, or whether overlapping runs are permitted.

Platform documentation makes the timezone assumption concrete. Google Kubernetes Engine’s CronJob documentation states that “all CronJob times are in UTC” and that the schedule uses five-field crontab format. Amazon EventBridge’s scheduled rules documentation states that “all scheduled events use UTC+0 time zone, and the minimum precision for a schedule is one minute.” If the author wrote the schedule thinking in local time and the platform interprets in UTC, the handoff should say so — and the next person should verify the actual fire times rather than trust the expression.

Overlap policy is the other assumption that bites. GKE’s documentation notes that if startingDeadlineSeconds is empty, the CronJob never times out, “which could lead to the same CronJob running multiple times simultaneously,” and that concurrencyPolicy defaults to Allow, meaning concurrent jobs are permitted. A homegrown job on a bare-metal crontab has no such policy field; overlap prevention is whatever the script does, or does not do. The handoff should state which it is.

Expected duration matters for the same reason. If a normal run takes four minutes and the schedule fires every five, a slow run overlaps. If a normal run takes forty minutes and the schedule fires hourly, the margin is thin. Write down the observed duration range, not the author’s recollection of it.

Write the runbook as decision points, not a command list

A list of commands tells the next person what to type. It does not tell them when to type it, when to stop, or when to escalate. Runbook guidance from the operations literature converges on the same structure: purpose, scope, trigger conditions, prerequisites, step-by-step instructions, validation, rollback, escalation, and revision history. For a cron job handoff, the decision points are the part that matters most.

For each failure mode, the runbook should answer:

• What do I check first, and what does each outcome tell me?
• When is it safe to re-run, and when will a re-run make things worse?
• What is the stop condition — the point at which I should stop investigating and escalate?
• Who do I escalate to, and what do they need to know?

The re-run question deserves special attention. A job that is idempotent can be re-run freely; a job that appends or increments cannot. The handoff should state which it is, and if the answer is “it depends on where it failed,” the runbook should say where.

Verify the backup and restore path the job depends on

If the job writes data that matters, the handoff is incomplete without a tested restore path. This is where lean teams most often discover, during a handoff, that the backup was never verified.

The tools have verification commands, and the handoff should name them. restic’s documentation describes a check command for checking repository integrity and consistency. pgBackRest’s user guide explains that incremental backups depend on prior backups being valid to restore — “all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup” — which means a handoff that documents only the latest backup is documenting an incomplete recovery path. The same guide notes that local and remote pgBackRest versions must match exactly, or WAL archiving and backups will not function; a version mismatch discovered during a restore is a bad time to learn it.

The handoff should include a restore rehearsal, not just a restore procedure. The rehearsal can be small: restore the most recent backup to a scratch instance, run the success check against it, and record the result. A procedure that has never been executed is a hypothesis.

Prefer a guard over a warning

The SRE Workbook’s critique of a bad postmortem includes an action item to “train humans not to run unsafe commands,” and the workbook’s response is direct: “In general, trying to change human behavior is less reliable than changing automated systems and processes.” The same principle applies to the handoff. If a failure mode is likely, the handoff should include a proposed guard — a lock file, a pre-flight check, an alert on the success check — rather than a note that says “be careful.”

A guard is also easier to verify. A note that says “do not run this twice” depends on the next person reading the note. A lock file that prevents a second run depends on nothing. Where the author can add the guard before leaving, that is worth more than a paragraph describing the hazard.

What the handoff document should contain

A workable structure, adapted from the runbook and postmortem guidance above:

1. Purpose and scope. What the job does, what it does not do, and which environments it runs in.
2. Schedule and assumptions. The expression, the timezone it is interpreted in, observed duration range, overlap policy, and what happens when a run is late or skipped.
3. Failure inventory. Each failure mode, how it is detected, what it damages, and the first check.
4. Success check. The independent verification that the job did its work, runnable by someone who did not write it.
5. Blast radius. Tables, buckets, queues, and downstream consumers touched, with the write pattern for each, and what is explicitly not touched.
6. Runbook. Decision points for each failure mode: first check, re-run safety, stop condition, escalation path.
7. Backup and restore. The verification command, the restore rehearsal result, and the version-match requirement if applicable.
8. Owner and review date. One name, one date.

The document does not need to be long. It needs to be specific enough that the next on-call can act on it without calling the author, and honest enough to say where coverage is missing. A handoff that says “we do not currently detect the silent no-op case” is more useful than one that implies detection exists.

Frequently asked questions

How long should the handoff document be?

Long enough to cover the failure inventory and the runbook decision points, and no longer. For a single scheduled job, that is usually a few pages. Length is not the goal; the test is whether someone who did not write the job can diagnose and recover it without calling the author.

What if the author has already left?

Reconstruct from what exists: the script, the logs, the scheduler configuration, and the incident history. The failure inventory becomes a set of hypotheses to verify rather than a record of known behavior. Start with the success check, because without it you cannot tell whether the job is working now.

Should the handoff include a rewrite plan?

It can, as a separate section, but the rewrite should not substitute for the failure inventory. A rewrite scheduled for next quarter does not help the on-call engineer next week. Document the current job’s failure modes first, then propose the rewrite as an action item with an owner.

How does this relate to a broader recovery checklist?

The cron job handoff is a narrow instance of a broader problem: knowing what to do when something breaks and the person who knew is unavailable. A recovery checklist for the wider system is a useful companion, and the Recovery Checklist covers that ground. The cron job document should link to it rather than duplicate it.

What is the single most important section?

The success check. Without it, the next person cannot distinguish a quiet job from a broken one, and every other section of the document is harder to use. If time is short, write the success check first.

Four root vegetables sit in a row, each with a face: a carrot, a radish, a beet, an onion. Their roots are drawn down into a band of soil, visible as thin lines rather than hidden. Around them, in a ring of capital letters, the shirt reads: LET’S ROOT FOR EACH OTHER AND WATCH EACH OTHER GROW.

The pun is doing two jobs at once, and it is worth separating them. To root for someone is to support them from the sidelines — cheer, encourage, want them to succeed. To root, for a plant, is to establish below the surface, to take hold in soil and draw what is needed. The artwork commits to the second meaning literally: the roots are drawn, the soil is drawn, the vegetables are smiling. The first meaning is carried only by the phrase. That split is the interesting part, and it is also where the shirt becomes more than a garden joke.

What the drawing actually shows

The four vegetables are arranged with their roots visible in soil, which is a specific choice. A carrot drawn without its root is a cartoon; a carrot drawn with its root in dirt is a diagram of a plant doing the thing plants do. The faces are simple — eyes and a smile — and the overall composition is a closed loop: the words surround the vegetables, the vegetables sit in the soil, the soil holds the roots. Nothing in the design suggests competition between the four. They are not racing. They are not arranged by size. They are simply four different root vegetables, each with its own shape and its own root system, sharing the same band of soil.

That is a visual argument, not just a decoration. The phrase watch each other grow implies comparison — you can only watch growth if you can see change over time. But the drawing gives each vegetable a different form. A radish does not grow into a carrot. The growth being watched is not convergence toward one ideal shape; it is each plant becoming more fully itself. I read that as the design’s best idea, and it is an interpretation of the artwork, not a claim about what the illustrator intended.

Two kinds of rooting, and the difference that matters

In work culture, rooting for each other is usually shorthand for encouragement: slack in the channel, a thumbs-up on the pull request, a kind word after a rough incident. That is real and worth having. But the second meaning of rooting — establishing below the surface — is the one that actually sustains a small team. A root system is not visible from above. It is the unglamorous infrastructure: the runbook nobody reads until 3 a.m., the decision log that explains why the database is configured the way it is, the onboarding checklist that assumes the new hire will be on call in six weeks. None of that looks like cheering. All of it is what lets a person grow into the role without the whole thing collapsing.

The shirt’s phrase works because it holds both meanings without resolving them. Root for each other is the visible, social layer. Watch each other grow is the slower, underground layer. The artwork puts the roots in the soil where they belong — visible in the drawing, but only because the illustrator chose to show them. In an actual garden, you don’t see the roots. You see the leaves, and you infer the roots from whether the plant is thriving. The same is true of a team: you see the deploys and the incident reviews, and you infer the health of the underlying practices from whether things keep working.

The slogan-versus-promise problem

There is a difference between a slogan and a usable promise, and this shirt sits right on the line. A slogan is a sentence that sounds good and asks nothing of you. A usable promise names a behavior, a boundary, or a condition that someone can actually check. Let’s root for each other is closer to a slogan. Watch each other grow is closer to a promise, because watching implies attention over time, and attention is a cost.

For a technical team, the useful move is to convert the slogan into something checkable. What does rooting for each other look like on a Tuesday when the pager has gone off twice and the deploy is blocked? It looks like the person who is not on call reading the incident channel and asking whether the runbook step was clear, rather than waiting to be asked. It looks like the reviewer who says this will be hard to roll back instead of looks good to me. It looks like the offboarding checklist that gets run even when the person leaving is well-liked and the team is busy. Those are roots. They are not visible from above, and they are the reason the plant stays upright.

The shirt does not claim any of this. It is a garden pun with smiling vegetables. But the pun is well-built, and the artwork gives it enough structure to think with. The four vegetables are different; the soil is shared; the roots are drawn. That is a better model for a team than a row of identical seedlings, and it is a better model than a slogan that asks for enthusiasm without asking for infrastructure.

Where the object fits

The shirt is a standard t-shirt from Quirky Cotton Collective, listed as Let’s Root For Each Other And Watch Each Other Grow Garden. The design is the phrase surrounding the four smiling root vegetables with their roots visible in soil. It is an off-duty object, not a tool, and it does not need to be more than that. But if you are the kind of team that keeps a decision log and a runbook, the shirt’s two meanings are a useful prompt: the visible encouragement is the easy part, and the underground work is the part that actually lets people grow.

What we would do differently: we would stop treating rooting for each other as a mood and start treating it as a set of underground practices that can be written down, assigned, and checked. The shirt is a reminder, not a procedure. The procedure is still ours to write.

Editorial note: this publication and Quirky Cotton Collective share an owner.

What a Blameless Postmortem Should Record When the Trigger Was a One-Line Terraform Typo

A one-line Terraform typo is a trigger, not a root cause. The postmortem that stops at the typo produces a fix as narrow as the mistake and leaves untouched the conditions that made the mistake consequential. For a team of two to fifteen engineers who are their own on-call rotation, the useful postmortem records the plan/apply gap, the state assumptions, the detection lag, and the missing guardrail — then converts those into bounded, owned action items.

This is a writing and analysis discipline, not a tooling purchase. The sources below describe general postmortem practice and Terraform’s documented behavior. They do not document a specific typo-triggered outage, and this article does not invent one. Where an example appears, it is labeled hypothetical.

Start by separating the trigger from the contributing causes

Google’s SRE book defines a postmortem as “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” The same chapter states that a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.”

Applied to a Terraform typo, that means the postmortem has at least two distinct sections:

• Trigger. The exact change: the file, the resource address, the diff, the commit hash, the plan output that was reviewed, and the apply output that ran. This is a factual record, not a judgment.
• Contributing causes. The conditions that allowed the change to reach production and to matter: what the plan showed, whether the applied plan matched the reviewed plan, what the team believed about state, how the change was detected, and what the recovery path required.

Google’s example postmortem template keeps these as separate fields — “Root Causes” and “Trigger” — which is a useful structural reminder that the trigger is one input among several. A postmortem that lists only the typo under root causes has collapsed the analysis into a single cause and will generate a single-cause fix.

Record the plan/apply facts, not the typo narrative

Terraform’s documentation is precise about what terraform plan does: it “creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure,” and “the plan command alone does not actually carry out the proposed changes.” The same page warns that “other changes made to the target system in the meantime might cause the final effect of a configuration change to be different than what an earlier speculative plan indicated, so you should always re-check the final non-speculative plan before applying to make sure that it still matches your intent.”

Those documented behaviors give the postmortem a factual frame. Record, with links or pasted excerpts:

• The speculative plan reviewed in the pull request, if any.
• The final non-speculative plan shown at apply time, and whether it matched the reviewed plan.
• Whether the apply used a saved plan file (-out=FILE) or generated a fresh plan at apply time.
• Whether -target was used. HashiCorp documents -target as intended for “exceptional circumstances, such as recovering from mistakes or working around Terraform limitations” and “not recommended … for routine operations, since this can lead to undetected configuration drift and confusion about how the true state of resources relates to configuration.” If the change used -target, that belongs in the record.

This section is where a lean team gets the most leverage. The typo is one character. The plan/apply gap is a process condition the team can change.

State assumptions belong in the postmortem

Terraform’s state documentation explains why state matters: “Terraform uses your workspace’s state to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. Terraform uses state to determine which changes to make to your infrastructure.” The same page recommends storing state in a remote backend with locking and access control, and warns against storing it in version control because that “can result in data loss or exposure of secrets stored in the state file.”

For the postmortem, the relevant questions are narrower:

• Was the state current at plan time? If a manual change had occurred outside Terraform, was it reconciled before the apply?
• Was state locking in effect, and did anyone else hold the lock during the apply?
• Did the change depend on a data source or module output that could have shifted between plan and apply?

These are conditions, not accusations. A team that records “state was stale because a manual security-group edit was never reconciled” has found something it can fix — a refresh step in the runbook, a drift check, a policy about manual changes during incidents. A team that records “engineer made a typo” has found nothing it can fix except the engineer, and Google’s guidance is explicit that “you can’t ‘fix’ people, but you can fix systems and processes.”

Capture the detection and response timeline with exact times

PagerDuty’s postmortem template asks for specific timestamps: “Time the impact began,” “Time of the page,” “Time that the status page was updated,” “Time of any significant actions,” and “Time the impact ended.” Google’s example postmortem uses a screenplay-style timeline with entries at one- or two-minute intervals.

For a lean team, the most useful derived number is the detection lag: the interval between the change landing and the first signal. Record whether that signal was a synthetic check, an alert, or a customer report. The action item that follows is often about detection, not about the typo.

Also record the response path: who was paged, how long until acknowledgment, what the first diagnostic step was, and whether the rollback or reconcile path was documented. If the team has no dedicated reliability coverage, note that as a condition — the absence of a second pair of eyes on the apply, the absence of an automated drift check — rather than as an excuse. The postmortem’s job is to make the condition visible so it can be scheduled.

Give blast radius and recovery their own section

PagerDuty’s template asks for exact numbers under Impact: “Time in SEV-1,” “Time in SEV-2,” “Events Dropped / Not Accepted,” “Accounts Affected,” “Users Affected.” Google’s workbook chapter on postmortem culture criticizes a bad example for providing only duration and no size or scope, noting that “a well-informed estimate is better than no data at all.”

For a Terraform-triggered incident, blast radius has a specific shape: which resources were created, changed, or destroyed; which services depended on them; and whether the change touched state that other workspaces or manual processes also manage. Record the recovery path in the same section: was mitigation a revert, a re-apply, a manual console change, a state edit, or a restore from backup? If recovery depended on a manual step that had not been rehearsed, that is a finding about operational readiness.

If the incident involved data loss or a database, the recovery section should name the backup and restore mechanism and whether it was tested. PostgreSQL’s documentation on continuous archiving and point-in-time recovery describes the mechanics — WAL archiving, base backups, replay to a target time — but the postmortem’s job is not to restate the manual. It is to record whether the team’s actual restore path was known, owned, and exercised. A restore lottery that has never been run is a condition worth writing down.

Write action items across categories, with owners and finish lines

Google’s USENIX article on postmortem action items classifies them by category — “Investigate, Mitigate, Repair, Detect, Prevent” — “to make sure that the action item plan covers both very short-term and longer-term fixes.” The same article requires that “every postmortem stemming from a user-visible event” have “at least one P0 or P1 action item,” and quotes Ben Treynor Sloss: “To our users, a postmortem without subsequent action is indistinguishable from no postmortem.”

PagerDuty’s template adds the wording discipline: “Write all action items to be actionable, specific, and bounded. Create tickets for all action items and label with their severity level and date tags.” Google’s article gives the contrast directly: “Investigate monitoring for this scenario” is poorly worded; “Add alerting for all cases where this service returns >1% errors” is better. “Fix the issue that caused the outage” is poorly worded; “Handle invalid postal code in user address form input safely” is better.

For a typo-triggered incident, a balanced plan might include:

• Detect: Add a synthetic check or alert that fires within N minutes when the affected resource becomes unreachable. Name the check and the threshold.
• Prevent: Add a required plan-diff review step to the change checklist, or a CI job that posts the plan output to the pull request. Name the file and the reviewer role.
• Mitigate: Document the rollback or reconcile procedure for this resource class in the runbook, with the exact commands and the expected output.
• Repair: Reconcile the state drift that the incident exposed, and add a drift check to the weekly maintenance routine.
• Investigate: Determine whether other workspaces share the same pattern, and record the result.

Each item needs a single owner, a ticket, a priority, and a finish line. “Be more careful with Terraform” is not an action item. “Add a plan-diff comment to the pull request template by [date], owned by [role]” is.

Keep the language blameless and the data specific

Google’s workbook chapter catalogs the failure modes of a bad postmortem: missing context, omitted key details, empty recovery sections, action items without owners, finger pointing, animated language, and delayed publication. Its examples of animated language — “careless ignorance,” “which is ridiculous,” “I can’t believe we survived this one!!!” — are worth keeping in mind on a small team where everyone knows who applied the change.

Blamelessness here is a writing discipline, not a disclaimer. Remove individual names from causal sentences. Replace adjectives with measurements. Keep the timeline factual. If the postmortem needs to say that a review step was skipped, say which step and what the checklist required, not who skipped it.

Publish quickly. PagerDuty’s template recommends scheduling the postmortem meeting “within 5 business days of any event.” Google’s workbook criticizes a postmortem published four months after the incident, noting that in the interim key details are forgotten. A short, honest postmortem published within days beats a comprehensive one published months later. For a lean team, the review meeting and the action-item tickets are the durable artifacts.

A hypothetical, clearly labeled

Suppose a team applies a one-line change to a security-group rule. The pull request showed a speculative plan with one rule change. At apply time, the engineer ran terraform apply without a saved plan, and Terraform generated a fresh plan. Because a manual console change from the previous week had never been reconciled, the fresh plan included an additional replacement of a database subnet group. The apply succeeded. The first signal was a customer report ninety minutes later.

The useful postmortem findings are not the typo. They are: the unreconciled state, the absence of a saved-plan step in the runbook, the missing drift check, and the ninety-minute detection lag. The action items follow from those findings. This example is constructed to illustrate the structure; it is not a record of a specific incident.

A short checklist for the typo-triggered postmortem

• Separate trigger from contributing causes in the document structure.
• Paste or link the reviewed plan and the applied plan; note whether they matched.
• Record state assumptions: currency, locking, drift, data sources.
• Record exact times for impact start, first signal, page, mitigation, and impact end.
• Record blast radius with numbers or a well-informed estimate.
• Record the recovery path and whether it was rehearsed.
• Write action items across investigate, mitigate, repair, detect, and prevent; give each a single owner, a ticket, a priority, and a finish line.
• Remove individual names from causal sentences and replace animated language with measurements.
• Publish within days, not months.

What to leave out when evidence is thin: any claim about how common this class of typo is, any statistic about detection latency or action-item closure in small teams, and any assertion that a specific review tool would have prevented the incident. The sources describe general practice; they do not establish causal effects for this scenario. Record what happened, label what is inferred, and let the action items carry the weight.

For the recovery side of the same discipline — writing the checklist before the incident, so the postmortem can record whether it was followed — see Write the Recovery Checklist Before You Need It.

Sources

• Google SRE Book, Chapter 15, “Postmortem Culture: Learning from Failure”: https://sre.google/sre-book/postmortem-culture/
• Google SRE Book, Appendix D, “Example Postmortem”: https://sre.google/sre-book/example-postmortem/
• Google SRE Workbook, Chapter 10, “Postmortem Culture: Learning from Failure”: https://sre.google/workbook/postmortem-culture/
• Lunney, Lueder, and Beyer, “Postmortem Action Items: Plan the Work and Work the Plan,” USENIX login:, Spring 2017: https://sre.google/static/pdf/login_spring17_09_lunney.pdf
• PagerDuty Postmortem Template: https://postmortems.pagerduty.com/assets/pdf/PostmortemTemplate.pdf
• HashiCorp, terraform plan command: https://developer.hashicorp.com/terraform/cli/commands/plan
• HashiCorp, Terraform state: https://developer.hashicorp.com/terraform/language/state
• PostgreSQL Documentation, “Continuous Archiving and Point-in-Time Recovery (PITR)”: https://www.postgresql.org/docs/current/continuous-archiving.html

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner. This is not an independent review.

The artwork is a stack of two messages. On top, a white rounded banner with a small bell icon reads There is motion at your front door. Below it, inside a circular frame tinted green, an orange tabby stares straight into the lens, close enough that its face fills the view. Porch siding and a light fixture sit behind it. The banner is the kind of alert a doorbell camera pushes to a phone; the circle is the camera feed that produced it. The joke is not that a cat is at the door. The joke is that the system did exactly what it promised and the human still has to decide what the event means.

That is a useful object for anyone who carries a pager, and not because it teaches a lesson about monitoring. It is useful because it gets one detail right that a lot of alerting prose gets wrong: it names the event without naming the threat.

What the banner actually says

Read the words literally. There is motion at your front door. Not someone is at your door. Not a package has arrived. Not your home is under attack. Motion. The banner reports a sensor reading and a location. It does not tell you who or what caused the reading, whether the reading is welcome, or what you should do next.

That restraint is the whole design. The camera view below supplies the interpretation the banner withholds. The banner is the alert; the cat is the context. Separated, each is thin. Together they form a complete incident: a trigger fired, and here is the evidence that lets you classify it.

We keep coming back to this because the failure mode it avoids is common in our own runbooks. An alert that says database is down has already made a judgment the monitoring system is not entitled to make. An alert that says PostgreSQL primary stopped accepting connections at 02:14 UTC; replica lag 0; last successful checkpoint 02:09 reports what happened and leaves the classification to the person holding the pager. The first phrasing invites a wrong action. The second invites a look.

The green tint is doing work

The circular view is tinted green, the way night-vision and low-light camera feeds often render. We are not claiming the design references any specific doorbell product; the observation is only that the tint reads as camera feed rather than photograph. That distinction matters to the joke. A photograph of a cat on a porch would be a picture of a cat. A green-tinted circle with porch siding behind it reads as a frame captured by a device, at a moment, because something moved.

The cat’s face is very close to the lens and slightly wide-eyed. We read that as the animal having approached the camera, not as fear or aggression. The design does not need the cat to be doing anything in particular. The cat only needs to be the thing that tripped the sensor. The humor comes from the gap between the banner’s neutral phrasing and the specific, slightly absurd cause.

A slogan versus a usable promise

There is a difference between a phrase that sounds like an alert and a phrase that could function as one. There is motion at your front door could function as an alert. It has a subject, a location, and a trigger condition. It does not have a severity, an owner, or a recommended action, which is why it is a notification and not a runbook step. The shirt is funny because it stops at the notification.

We have written before about shirts that carry a sentence and ask the reader to complete it. This one is different. The sentence is complete. What is missing is the decision layer, and the design deliberately withholds it. The cat is not a punchline appended to the alert. The cat is the reason the alert is insufficient on its own.

If you want the object itself, the listing is here: There Is Motion At Your Front Door Funny Cat Doorbell Meme. The design is a standard t-shirt print; the visual observation above is what we are describing, not a claim about fabric, fit, or how it wears over time.

What the design does not claim

It does not claim the cat is a threat. It does not claim the camera is smart. It does not claim the homeowner should panic or relax. The banner is neutral and the image is specific, and the tension between those two registers is the entire piece. We think that is why it works as an off-duty object for people who spend their working hours writing and reading alerts. It is a small, accurate parody of the moment before classification, when the pager has fired and you have not yet decided what kind of event you are in.

We are not going to claim the shirt improves anyone’s incident response. It is a print of a notification and a cat. The observation we are willing to make is narrower: the wording is disciplined, and the image supplies the context the wording refuses to invent. That is a better division of labor than most of the alerts we have written.

What we’d do differently

If we were adapting this design’s logic to our own alerting, we would steal the banner and leave the cat. Name the sensor, name the location, name the time, and stop. Put the interpretation in the dashboard or the runbook link, not in the alert text. The shirt gets this right by accident of comedy. Our PagerDuty services get it wrong by habit. The next time we edit an alert, we will ask whether the message reports an event or announces a conclusion, and if it announces a conclusion, we will cut it back to the event.

Editorial note: This publication and Quirky Cotton Collective share an owner. This is not an independent review.

The shirt reads I WEAR THIS SHIRT PERIODICALLY, and beneath that sentence sit four tiles in the visual grammar of the periodic table: 53 I, 74 W, 22 Ti, 16 S. The tile symbols are I, W, Ti, S. Read straight across, they do not spell the sentence. They spell I W Ti S — a fragment that only resolves when the reader supplies the missing letters from the sentence above and hears the pun land: periodically, as in the periods of the table, as in now and then.

That gap between what the tiles say and what the sentence says is the whole design. It is also the most interesting thing about the shirt, and the thing worth slowing down on. A lesser version of this joke would have printed the sentence and stopped, trusting the word periodically to carry the chemistry reference alone. This one makes the reader do a small piece of assembly work. The tiles are not decoration; they are a second, incomplete text that the first text completes. The joke is not the pun. The joke is the act of decoding.

What the tiles actually claim

Each tile follows the same pattern: atomic number on top, element symbol below. 53 is iodine, symbol I. 74 is tungsten, symbol W. 22 is titanium, symbol Ti. 16 is sulfur, symbol S. Those are the standard atomic numbers and symbols for those elements, and the tiles are arranged left to right in that order. The design is not a scrambled periodic table or a set of invented elements. It is four real entries, borrowed for their symbols.

What the tiles do not do is spell the word they sit under. The sentence says periodically. The tiles say I W Ti S. To get from one to the other, you have to notice that the sentence’s first word is I, that W can stand in for the sound at the start of wear, that Ti can be read as the opening of this, and that S can be read as the opening of shirt. The tiles are not a phonetic transcription. They are a set of prompts, and the sentence is the answer key printed directly above them.

This is a different kind of shirt slogan from the ones that simply assert a personality. It does not say I am a chemist or trust me, I’m a scientist. It says something narrower and stranger: I wear this shirt periodically. The adverb is doing double duty. In ordinary English it means from time to time. In the presence of the tiles it also means according to the periods of the periodic table. The shirt is a sentence that becomes a different sentence once you have read the artwork beneath it.

The difference between a slogan and a usable promise

We spend a lot of working hours around terse messages that are supposed to coordinate behavior: runbook headers, alert titles, escalation notes, the one-line summary at the top of a postmortem. The recurring failure mode is not that these messages are wrong. It is that they are underspecified — they read as complete sentences while leaving the reader to guess which part of the sentence is load-bearing.

This shirt is a useful small example of the opposite. The sentence I wear this shirt periodically is grammatically complete and semantically thin. On its own it is a slogan: a mood, a bit of self-description, nothing you could act on. The tiles change that. They add a second layer that is incomplete on purpose and that points back at the sentence for completion. The reader who does the work gets the joke; the reader who does not still gets a wearable sentence. Neither reader is misled. The design does not pretend the tiles spell something they don’t.

That is a higher bar than it sounds. A lot of technical writing fails precisely here: it presents a partial diagram or a truncated command as if it were the whole procedure, and the reader who does not already know the missing step is left to invent it. The shirt’s tiles are honest about being partial. They are labeled with real atomic numbers and real symbols, arranged in a legible order, and they do not claim to be a word. The sentence above them is the context that makes the fragment meaningful. Strip the sentence away and the tiles are just four elements. Put the sentence back and they become a joke about recurrence.

Why the pun survives the second reading

Most pun shirts are one-and-done. You read them once, you get it, and the garment becomes a background object. The periodically design has a little more staying power because the decoding is not instantaneous. The eye goes to the sentence, then to the tiles, then back to the sentence. The tiles are not arranged to spell periodically; they are arranged to spell the first four content words of the sentence, and only approximately. That approximation is the joke. It rewards a reader who is willing to sound things out.

There is also a quieter layer: the word periodically is a chemistry word before it is a scheduling word. Periods are rows of the periodic table. To wear the shirt periodically is, in the design’s own logic, to wear it in accordance with the table — which is to say, in accordance with the four elements printed on it. The shirt is not just making a pun on the adverb. It is proposing a tiny rule for its own wearing. That is a more interesting object than a shirt that simply announces a hobby.

We would not overclaim what this demonstrates. It is a piece of printed apparel, not a lesson in pedagogy or a case study in interface design. But it is a clean example of a message that separates its surface reading from its intended reading without deceiving either audience. The sentence works as a sentence. The tiles work as a puzzle. The two together work as a joke. Nothing in the design requires the reader to pretend the tiles spell a word they don’t.

What we’d do differently

If we were writing the product copy rather than reading the artwork, we would resist the temptation to describe the tiles as spelling periodically. They don’t. They spell I W Ti S, and the sentence above them supplies the rest. The accuracy matters because the joke depends on the gap. A description that closes the gap for the reader — the elements spell out the sentence! — removes the small act of decoding that makes the shirt worth a second look. The better copy would point at the tiles, name the four elements, and let the reader hear the pun. The shirt is available at the Quirky Cotton Collective listing if you want to read the arrangement yourself.

The other thing we would do differently is resist the urge to turn this into a lesson about communication in general. The shirt is a specific object with a specific design, and its interest is local: four element tiles, one adverb, one sentence that completes the other. The general principle — that a partial message can be honest if it points clearly at its context — is real, but it is not what the shirt is about. The shirt is about the pleasure of reading something twice and getting a different sentence the second time. That is enough.

The AMI Pin Is a Decision: Write It Down Before the Next Incident

A service is still launching from an AMI that predates the current launch template by two years. The instance type has changed, the security groups have changed, the user-data script has been rewritten twice. The AMI ID has not moved. Nobody on the current rotation remembers why, and the ticket that might have explained it was closed without a comment.

This is not a story about a bad AMI. It is a story about a missing decision record. The pin itself may be entirely correct. The problem is that the next engineer to touch the launch configuration has no way to tell whether moving the AMI is routine maintenance or a change that will break something they cannot see.

Why a ticket comment is not enough

An architectural decision record (ADR) is a short document that describes a choice the team makes about a significant aspect of the software architecture, including its context and its consequences. AWS Prescriptive Guidance defines it that way and notes that the structure focuses on the reason for the decision rather than how the team implemented it (AWS Prescriptive Guidance, ADR process). That distinction matters here. A ticket comment usually records the implementation — “pinned to ami-0abc because the build failed” — and loses the reason. Six months later, the build failure is gone and the reason is unrecoverable.

The same guidance says that at a minimum, each ADR should define the context of the decision, the decision itself, and the consequences for the project and its deliverables. That is a low bar. It is also exactly the bar a lean on-call team needs: three sections, one page, stored where the launch configuration lives.

What the AMI lifecycle actually does

Before writing the record, it helps to be precise about what a pin protects against and what it does not. AWS documents that deprecated AMIs are hidden from AMI listings by default, but existing users and launch services such as launch templates and Auto Scaling groups can continue to use a deprecated AMI by specifying its ID (Deprecate an Amazon EC2 AMI). Instances launched from an AMI that is later deprecated are not affected and can be stopped, started, and rebooted. Deprecation does not delete the AMI, and the owner continues to pay for its snapshots until the AMI is deregistered.

Two consequences follow. First, a pin can outlive the deprecation date silently, because the launch template keeps working. Second, the pin is not free: the snapshots behind the AMI remain billable until someone deregisters it. Neither fact tells you whether the pin is safe. Both facts belong in the record as review triggers.

An AMI is also specific to region, operating system, processor architecture, root volume type, and virtualization type (Amazon Machine Images in Amazon EC2). A pin that was correct in one region may not be portable to another without a copy step, and a pin that was correct for one instance family may not be valid for a replacement. The record should name the scope of the pin, not just the ID.

A minimal ADR skeleton for an AMI pin

The following template is deliberately short. It is meant to be written in the same pull request that changes the launch template, or in the same week the pin is discovered. It is not a design document.

# ADR-00X: Pin service-name to AMI ami-xxxxxxxxxxxxxxxxx

## Status
Proposed | Accepted | Superseded by ADR-00Y

## Date
YYYY-MM-DD

## Context
- Service: service-name (repo, environment, region)
- Launch mechanism: launch template lt-xxxx / Auto Scaling group asg-xxxx
- Current AMI: ami-xxxxxxxxxxxxxxxxx, created YYYY-MM-DD
- What forced the pin: (one or two sentences, specific)
- What was tried and rejected: (optional, one line each)
- What we do not know: (state it plainly)

## Decision
Launch service-name from ami-xxxxxxxxxxxxxxxxx in region rrrr.
Scope: launch template lt-xxxx only. Other services are not covered.

## Consequences
- Moving the AMI requires: (named check, named owner)
- If the AMI is deprecated: launch template continues to work; review still required
- Snapshot cost continues until deregistration
- Related runbook: (link)
- Related restore drill evidence: (link, if any)

## Review trigger
- AMI deprecation date: YYYY-MM-DD (from describe-images)
- Review owner: name or rotation
- Next review: YYYY-MM-DD or on trigger, whichever first

Two fields carry most of the weight. The context field is where the reason lives, and the review trigger field is what keeps the record from becoming archaeology. The rest is bookkeeping.

Keep the record honest about what it does not know

If the original reason for the pin is lost, say so in the context section. Write “the reason for the original pin is not recorded; this ADR captures the current state and schedules an investigation” rather than inventing a plausible cause. A record that admits ignorance is more useful than one that fabricates certainty, because the first review can then be scoped as an investigation rather than a ratification.

The same applies to resilience claims. A pinned AMI is not a backup, and a decision record is not a restore drill. If the service depends on a database, the record should link to the restore evidence separately rather than implying that the pin provides it. The recovery checklist is the right companion document for that side of the service; the ADR covers the image, not the data.

Proposed, accepted, superseded

AWS Prescriptive Guidance is explicit about lifecycle. When the team accepts an ADR, it becomes immutable. If new insights require a different decision, the team proposes a new ADR, and when the new one is accepted it supersedes the previous one. The best-practices page adds that the owner should change the old ADR’s status to Superseded, note the changes in the new ADR’s change history, and keep the old ADR in the decision log (Best practices for using ADRs).

For a small team, this means one rule: do not edit an accepted pin record to reflect a new AMI. Write a new record, mark the old one Superseded, and link them. The old record is the audit trail that explains why the service ran on an old image for as long as it did. Editing it destroys the only evidence a future reviewer would have.

The Proposed state is also useful. A pin written under incident pressure can be marked Proposed, reviewed in ten to fifteen minutes at the next team sync, and then accepted or reworked. The AWS guidance suggests that a dedicated reading slot of that length is usually enough for review. That is a realistic commitment for a team of two to fifteen engineers without a dedicated reliability function.

Review triggers drawn from AMI lifecycle signals

A review trigger should be tied to evidence the team can actually observe. Three signals are available from AWS documentation and the console:

• Deprecation date. The deprecation time is visible in the console and returned by describe-images as DeprecationTime. If the field is empty, the AMI is not deprecated or is set to deprecate at a future date. The record should carry the date, not a vague “review periodically.”
• Last-used check. AWS notes that AMIs not used recently to launch an instance might be good candidates for deprecation or deregistration, and points to a last-used check. A pin that has not launched an instance in a year is a different decision than one that launches every deploy.
• Failed restore or failed launch. If a restore drill or a canary launch from the pinned AMI fails, that is a review trigger regardless of the calendar.

These signals do not tell the team whether to move the AMI. They tell the team when to look. The decision record supplies the context for that look.

What the record should not pretend to know

A decision record is not a security assessment. It should not claim that a pinned AMI is compliant, supported, or safe. The retrieved AWS documentation describes deprecation and lifecycle mechanics; it does not establish that any particular pin is appropriate. If the team has a vulnerability scan or a patch-level inventory, link it. If not, say that the record does not cover it.

Nor should the record claim a maximum safe duration for a pin. No source supports a specific number, and inventing one would turn a useful document into a false assurance. The honest formulation is: the pin remains in place until a named trigger fires or a named owner reviews it, and the record states both.

A checklist to paste into the repo

The smallest sustainable version of this practice is a checklist in the repository that owns the launch template:

1. Does an ADR exist for the current AMI pin? If not, write one this week using the skeleton above.
2. Does the context section name the forcing reason, or state plainly that it is unknown?
3. Does the decision section name the AMI ID, region, and launch mechanism scope?
4. Does the consequences section link to the runbook and to any restore drill evidence?
5. Does the review trigger carry a deprecation date and a named owner?
6. When the pin changes, is the old ADR marked Superseded rather than edited?

None of this requires a new tool. A Markdown file in the same repository as the launch template, reviewed in the same pull request, is enough. The point is not ceremony. The point is that the next engineer to open the launch configuration can read one page and know whether they are doing maintenance or making a decision.

Questions that come up

Is an AMI pin an architectural decision?

AWS Prescriptive Guidance lists dependencies and construction techniques among the decisions that warrant an ADR. A base image is a dependency and a construction technique at once. Whether a team treats every pin as architecturally significant is a judgment call, but a pin that has survived multiple launch-template revisions and whose rationale is unknown is a reasonable candidate.

What if the team has no ADR process at all?

Start with one record for the pin that is currently causing the most uncertainty. The AWS guidance recommends storing ADRs in a central location, either a Git repository for versioning or a wiki for accessibility. For a small team, the repository that owns the launch template is usually the lowest-friction choice.

Does deprecation force a migration?

No. AWS documents that launch templates and Auto Scaling groups can continue to reference deprecated AMIs, and that instances launched from a deprecated AMI are not affected. Deprecation is a signal, not a deadline enforced by the platform. The record should treat it as a review trigger, not an automatic change.

Should the record include the AMI creation date and patch level?

Yes, where known. The creation date is available from the AMI metadata. Patch level is only meaningful if the team records it at build time; if it is not recorded, the context section should say so rather than guess. The record’s value depends on distinguishing what was observed from what was assumed.

How does this relate to backup and restore drills?

They are separate decisions with separate evidence. The AMI pin governs what software boots; the restore drill governs whether data can be recovered. Link them, but do not let one stand in for the other. A service can have a perfectly documented AMI pin and an untested restore path, or the reverse.

The pin will be revisited eventually, whether by a planned review or by an incident. The decision record determines which of those it is.

Three elements, in order: a yellow caution triangle, a hand with a diagonal slash through it, and the words NOT FRIENDLY DO NOT TOUCH set in the block lettering of a hazard placard. The triangle is the first thing the eye finds, because yellow on a dark field is the highest-contrast shape on the garment. The crossed-out hand is second. The text is third, and it is the only part that is actually a sentence.

That ordering matters. A slogan shirt usually leads with the words and treats the graphic as decoration. This design inverts the hierarchy: the icon does the warning, and the text is the caption. It borrows the grammar of a physical sign — the kind bolted to a door, a fence, a piece of equipment — and moves it onto a torso. The result is not a punchline so much as a portable placard.

We keep coming back to that distinction because it is the difference between a shirt that says something and a shirt that does something. The not friendly do not touch shirt is, in our reading, an attempt at the second kind. It is a boundary expressed in the visual language of industrial safety, and the interesting question is whether that language survives the move from a locked cabinet to a human being.

What the sign format promises

A hazard sign is a contract. It says: this object is dangerous, this action is prohibited, and the prohibition is enforced by something — a lock, a fence, a supervisor, a law. The sign is not the enforcement. It is a legible proxy for enforcement that already exists. When you see a yellow triangle on a substation door, you do not test the door to see whether the sign is serious. The sign is serious because the institution behind it is serious.

On a shirt, there is no institution. There is a person, and the person has chosen to wear the sign. That is a different kind of claim. It is not a warning backed by force; it is a warning backed by preference. The crossed-out hand is doing the work of a fence, but the fence is social, not physical. The design is honest about this in a way that a purely verbal shirt would not be: by using the sign format, it admits that it is a proxy, and it asks the reader to treat the proxy as if it were real.

Whether that works is not something we can measure. What we can say is that the design is unusually coherent about its own limits. It does not say danger. It says not friendly, which is a softer and more specific claim. It does not say keep out. It says do not touch, which is a narrower prohibition. The wording is calibrated to the actual boundary a person might want to set in a shared space, not to the maximum alarm a sign could produce.

The crossed-out hand is the load-bearing element

Strip the hand away and you have a yellow triangle and some text. That is a mood. The hand is what makes it a rule. A crossed-out symbol is a prohibition in the imperative mood — it does not describe, it forbids — and it is one of the few graphic conventions that reads the same way to almost everyone who has been in a public building. The diagonal slash is the universal negation operator of the physical world.

We noticed that the hand is not stylized. It is a flat pictogram, the kind you would find on a fire exit or a no-smoking placard. That flatness is doing something important: it removes the hand from the realm of the personal. A photograph of a hand with a slash through it would read as violent. A pictogram reads as administrative. The design wants the prohibition to feel procedural rather than emotional, and the pictogram is how it gets there.

This is the same reason the text is set in warning-sign lettering rather than a friendly sans-serif. The type is not trying to be charming. It is trying to be legible at a distance, which is what hazard signage is for. The shirt is designed to be read across a room, not studied up close, and every choice — the yellow, the triangle, the block letters, the pictogram — serves that goal.

What a boundary shirt can and cannot do

There is a real difference between a slogan and a usable promise, and it is the difference between a shirt that states a preference and a shirt that changes an interaction. A slogan is a claim about identity: I am this kind of person. A usable promise is a claim about behavior: this is what will happen if you do that. The warning-sign format pushes toward the second category, but it cannot complete the move on its own. The shirt can announce the boundary. It cannot enforce it.

That is not a flaw in the design. It is a limit of the medium, and the design seems to know it. The phrase not friendly is doing something subtle here. It is not a threat; it is a disclosure. It tells the reader what to expect, which is a more honest move than pretending the shirt is a force field. The crossed-out hand is the rule, and the text is the explanation of the rule, and neither of them claims to be the enforcement.

We think this is why the design reads as a boundary rather than a joke. A joke would undercut the warning — it would signal that the sign is not serious, that the wearer is in on the gag. This design does not wink. The yellow is the same yellow as a real caution sign. The lettering is the same lettering. The only thing that marks it as a shirt rather than a placard is the fabric, and the fabric is the one element the design cannot control.

The off-duty object as a legible artifact

We spend most of our working hours on runbooks, checklists, and decision records — documents whose entire purpose is to make a boundary or a procedure legible to someone who was not in the room when it was decided. A good runbook does not rely on the reader already knowing what to do. It states the rule, states the trigger, and states the consequence, in that order, in language that survives being read at 3 a.m. by someone who is tired and scared.

The warning-sign shirt is a small, low-stakes version of the same problem. It is trying to make a boundary legible to a stranger, in public, without a conversation. The triangle is the trigger. The crossed-out hand is the rule. The text is the consequence, softened to a disclosure. It is not a runbook, and we would not pretend it demonstrates anything about operational practice. But it is a legible artifact, and legibility is a craft we care about.

The tradeoff is the same one we make in documentation: the more explicit you are, the more you commit to a specific reading. A vague shirt can mean anything. This shirt means one thing, and it means it in a format that most people already know how to read. That is a cost — it forecloses the possibility of being read as playful — and it is also the point. The design has chosen clarity over ambiguity, and it has chosen a visual vocabulary that carries its own authority.

What we’d do differently

If we were designing a boundary artifact for a small team — a sign for a shared lab, a status board, a do-not-disturb marker for a focus room — we would steal the hierarchy from this shirt and leave the rest. Icon first, rule second, explanation third. The icon carries the prohibition, the rule makes it specific, and the explanation softens it enough to be usable in a shared space. What we would not do is rely on the artifact to enforce itself. A sign is a proxy for a norm, and a norm only holds if the people around it agree to treat the proxy as real. The shirt is a good example of a design that knows the difference.

Editorial note: this publication and Quirky Cotton Collective share an owner.