Gray Haven Lab

Editorial note: This publication and Quirky Cotton Collective share an owner.

The shirt reads I WEAR THIS SHIRT PERIODICALLY, and beneath the sentence sit four tiles in the visual grammar of the periodic table: 53 I, 74 W, 22 Ti, 16 S. The symbols are I, W, Ti, S. Read straight across, they do not spell periodically. They spell I W Ti S — a near-miss that is the entire joke, and the reason the design is worth a closer look than most slogan shirts get.

We keep a small decision log of visual jokes that land because of what they refuse to do. This one refuses to be a rebus. A rebus would hide the word and ask you to solve it. This design puts the word in plain English at the top, then offers a second layer that almost resolves into the same word and stops one letter short. The pleasure is in the gap.

What the tiles actually say

Four tiles, four atomic numbers, four symbols. The numbers are real: 53 is iodine, 74 is tungsten, 22 is titanium, 16 is sulfur. The symbols are real too. What is not real is the sentence they form. I W Ti S is not a word. It is a sequence that a chemistry-literate reader will try to pronounce, fail to pronounce, and then recognize as a visual pun on the word printed above it.

That is a different kind of joke from the one the retail title implies. The title says “Funny Chemistry Science Pun,” which is accurate but flat. The design is funnier than the title because it is more restrained. It does not add a caption explaining the gag. It does not draw an arrow. It trusts the reader to notice that the tiles are arranged like elements and to hear the echo of periodically in I W Ti S without being told.

We noticed the same restraint in the I Wear This Shirt Periodically chemistry element-tile pun listing itself: the artwork is described by its parts, not by a claim about what wearing it proves. That is the right level of promise for a shirt. It is a diagram joke, not a credential.

The difference between a slogan and a usable promise

Technical teams spend a lot of time around messages that are supposed to do work: runbook steps, alert text, status-page wording, on-call handoff notes. A slogan and a usable promise can look similar on a shirt or a slide. The difference is whether the words survive contact with a reader who has to act on them.

I wear this shirt periodically is a slogan. It is true in a narrow, literal sense — the wearer puts it on from time to time — and it is funny because the periodic-table tiles make the adverb visible as a set of elements. But it does not tell you when, why, or what to do. It is not a schedule. It is not a rotation policy. It is a joke about the word periodically, and it works precisely because it does not pretend to be more.

We have written before about warning signs that name their own failure mode. This shirt is the opposite: a message that names its own limitation. The tiles do not spell the word. The sentence does not promise a cadence. The design is honest about being a visual gag, and that honesty is what makes it wearable in a room full of people who read diagrams for a living.

Why the near-miss matters

If the tiles spelled periodically exactly, the shirt would be a puzzle with a solution. You would read it once, get it, and move on. The near-miss keeps the joke alive a half-second longer. It also creates a small social test: the reader who knows the element symbols sees the gap immediately; the reader who does not sees four tiles and a word and may not notice anything is missing. Neither reader is wrong. The shirt simply rewards a particular kind of attention.

That is a useful thing to notice about technical communication in general. The messages that stick are often the ones that leave a small, deliberate gap for the reader to close. A runbook step that says “verify the backup” is weaker than one that says “verify the backup restores to a scratch instance and record the row count.” The second version gives the reader something to do with their attention. The shirt does the same thing at a much lower stakes: it gives the eye a sequence to resolve and then withholds the final resolution.

We are not claiming the design was built with that theory in mind. We are claiming that is how it reads. The observation is ours; the intent is not documented anywhere we can see.

What we would do differently

If we were writing the retail copy for this shirt, we would lead with the gap instead of the genre. “Funny Chemistry Science Pun” tells a shopper what shelf the shirt belongs on. It does not tell them why the shirt is funny. A line like “Four element tiles that almost spell the word above them” would do more work, and it would set a more accurate expectation for the reader who notices the near-miss and the reader who does not.

We would also resist the urge to add a caption to the artwork itself. The design is stronger without one. The gap is the joke. Filling it in would be the visual equivalent of explaining a runbook step that already works.

The shirt is available at the link above. It is a diagram joke, a boundary message, and a small lesson in restraint, all printed on the same garment. We would wear it periodically. We would not claim it proves anything about our on-call rotation.

A missing segment is a decision, not an error to clear

Point-in-time recovery stops on a missing WAL segment because the archive cannot supply a file PostgreSQL needs to continue replay. The PostgreSQL documentation is direct about the precondition: recovery using continuous archiving requires “a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup” (PostgreSQL 18, Continuous Archiving and PITR). When that sequence has a hole, the restore cannot reach the target you asked for. It can only reach a point before the hole, or start again from a different base backup.

That makes the missing segment a decision point. The runbook’s job is not to make the decision for the on-call engineer. It is to make the decision explicit, reproducible, and auditable after the fact. A runbook that says “investigate missing WAL” leaves the next reader to reconstruct intent under pressure. A runbook that records the target, the exact segment, the lookup that failed, the branch chosen, and the trade-off accepted gives the next engineer a starting position instead of a blank page.

This is written for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated SRE function. The structure below is a recommendation, not a standard. It is built around what the PostgreSQL and pgBackRest documentation actually exposes, plus the failure modes the Crunchy Data write-up on missing WAL files describes.

Record the recovery intent before the failure

The same missing segment means different things depending on what the restore was trying to reach. A segment missing on the path to a named restore point taken five minutes before a bad migration is a different problem from a segment missing on the path to a timestamp three days ago. Record intent first, so the failure has context.

At minimum, capture:

• Recovery target type and value. Whether the target is a timestamp, a named restore point, an LSN, or a transaction ID, and the exact value. If a named restore point was used, record the name and the command that created it.
• Target action. Whether recovery is expected to pause, promote, or shut down when the target is reached. The target action determines what the cluster does after replay, and it should be recorded alongside the target itself.
• Base backup or backup set. The backup label, the backup set identifier, and the repository it came from. pgBackRest’s user guide notes that “a restore requires the backup files and one or more WAL segments in order to work correctly” (pgBackRest User Guide). The backup set is half of that equation and should be named.
• Who requested the restore and why. One sentence. This is the part that gets lost first and matters most in the postmortem.

Recording intent first also protects against a subtle failure: a restore that succeeds to the wrong target. If the runbook only records the missing segment, a later reader cannot tell whether the restore that eventually completed actually reached the point the requester needed.

Record the exact failure, verbatim

Paraphrased errors are not reproducible. The runbook entry should contain the literal text PostgreSQL or the backup tool emitted, plus the invocation that produced it.

Useful fields:

• Missing segment name. The full 24-character segment name, including the timeline prefix. pgBackRest’s documentation describes the naming convention: the first eight hexadecimal digits are the timeline and the next sixteen are the logical sequence number (pgBackRest User Guide). The timeline matters because a failover or switchover increments it, and a segment that exists on one timeline may not exist on another.
• Verbatim error text. The Crunchy Data article lists several distinct messages that point at different problems, including ERROR: requested WAL segment 00000002000000050000007C has already been removed, INFO: unable to find 0000001600000000000000C2 in the archive, and FATAL: requested timeline 23 does not contain minimum recovery point 0/C2A56FC0 on timeline 22 (Crunchy Data, How to Recover When PostgreSQL is Missing a WAL File). These are not interchangeable. “Already been removed” suggests retention or recycling on the source. “Unable to find in the archive” suggests the archive lookup itself failed. The timeline message suggests a timeline mismatch. Copy the exact line.
• The restore_command or archive-get invocation. The exact command PostgreSQL ran to fetch the segment. With pgBackRest this is typically pgbackrest --stanza=<name> archive-get %f "%p", but the runbook should record what was actually configured, not what the template says. pgBackRest’s command reference notes that archive-get is “used by PostgreSQL to restore a backup, perform PITR, or as an alternative to streaming for keeping a replica up to date” (pgBackRest Command Reference).
• Timestamp and host. When and where the failure was observed, in UTC.

This is the part of the runbook most likely to be skipped during an incident and most valuable afterward. A verbatim error line lets the next reader search logs, reproduce the lookup, and compare against the archive’s own records.

Record the archive lookup path and its result

A missing segment is not the same as a missing archive. The runbook should distinguish between “the segment is not in the archive” and “the archive lookup did not run correctly.” Those lead to different branches.

Record:

• Which repository or archive location was queried. If pgBackRest is configured with multiple repositories, WAL is fetched “from the repositories in priority order (e.g. repo1, repo2, etc.)” (pgBackRest Command Reference). The runbook should name the repositories in priority order and record which one was reached before the failure.
• What the lookup returned. Whether the archive-get command returned a file, returned nothing, or errored. If it errored, the error text belongs here, not in the previous section.
• Whether asynchronous archive-get was in use. pgBackRest’s --archive-missing-retry option controls whether a segment previously reported missing in asynchronous mode is retried. The documentation notes that disabling it “allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary” (pgBackRest Command Reference). If the team uses asynchronous archive-get, the runbook should record whether retry was enabled and whether the spool path was cleared before the restore. This is a configuration detail that changes the meaning of a “missing” result.
• Whether the segment exists on a different timeline. If the error mentions a timeline, record which timelines were checked and what was found. A segment present on timeline 22 but requested on timeline 23 is a different problem from a segment absent everywhere.

The distinction matters because the branches differ. If the segment is genuinely absent from every repository and every timeline, retrying the lookup will not help. If the lookup failed for an operational reason, a retry may resolve it without touching the recovery target.

Record the decision branch and who owns it

This is the core of the runbook. When the segment is genuinely missing, the team has a small number of options, and each carries a trade-off that should be written down at the moment it is accepted.

The branches, as the documentation supports them:

1. Retry the archive fetch. Appropriate when the lookup failed for an operational reason rather than because the segment is absent. The runbook should record what was retried, how many times, and what changed between attempts. If the retry succeeds, record the segment that was recovered and the time it took.
2. Move the recovery target earlier, to a point covered by available WAL. This accepts additional data loss in exchange for completing the restore. The runbook should record the new target, the reason the earlier point was chosen, and what is known to be lost between the new target and the original one. The Crunchy Data article notes that PostgreSQL’s own hint in some recovery failures is to “choose an earlier recovery target” (Crunchy Data). That hint is a starting point, not a decision; the decision belongs to whoever owns the data.
3. Rebuild from a newer base backup. Appropriate when the gap is large enough that moving the target earlier would lose more than the team is willing to accept, or when the base backup itself is suspect. The runbook should record which backup set was chosen, why, and what the expected recovery point is.
4. Stop and escalate. A legitimate branch. If none of the above can be decided safely by the on-call engineer, the runbook should name who is called and what information they need. This is not a failure of the runbook; it is the runbook working.

For each branch, record:

• The decision owner. A role, not a name, so the runbook survives turnover.
• The trade-off accepted. One or two sentences. “Accepted loss of approximately four hours of writes between the original target and the earlier recovery point” is a record. “Chose earlier target” is not.
• The time the decision was made. In UTC, alongside the failure timestamp, so the postmortem can reconstruct the timeline.

The trade-off is the part most likely to be forgotten and most likely to be re-litigated. Writing it down at the moment of decision is cheaper than reconstructing it afterward.

Record post-recovery verification as a checklist, not a sentence

“Verify data” is not a verification step. The runbook should list what was compared, against what, and what remains uncertain.

Useful checks, in the order they are usually run:

• Recovery completed to the intended target. The cluster’s reported recovery target and the actual last replayed LSN or timestamp should be recorded. If the target was moved earlier, record both the original and the new target.
• The cluster is accepting connections and is not in recovery. If the target action was to promote, confirm the promotion happened. If the target action was to pause, confirm the pause is intentional and who will resume.
• Row counts or checksums for the tables the requester named. The requester should have named specific tables or queries before the restore started. If they did not, that is a gap to record, not to paper over.
• Sequence values and recent inserts. A restored cluster can be internally consistent and still be missing the most recent writes. Compare the highest sequence value or the most recent timestamp in a known table against the expected recovery point.
• What remains uncertain. If the restore reached an earlier target than requested, record what is known to be missing and what is unknown. “Writes between 14:00 and 18:00 UTC on the affected tables are not present; whether any of those writes were committed is not known from the archive alone” is a record. “Some data may be missing” is not.

The verification checklist should be written before the restore starts, not after. The requester is the best source of what “correct” means, and they are most available before the incident is resolved.

Record follow-up as a question, not a verdict

The follow-up section of the runbook should route to the blameless postmortem, not to a conclusion. The question is why the gap existed, and the answer is usually one of a small number of causes that the documentation describes.

Candidate questions to record:

• Did archiving fail repeatedly before the gap? PostgreSQL’s WAL configuration documentation notes that “if WAL archiving cannot keep up with the pace that WAL is generated, or if archive_command or archive_library fails repeatedly, old WAL files will accumulate in pg_wal until the situation is resolved” (PostgreSQL 18, WAL Configuration). Accumulation is visible; a gap that appears later may have started as an archiving failure that was not monitored.
• Did retention expire the segment before the restore needed it? The Crunchy Data article describes a replica that has been disconnected long enough for archived WAL to pass its retention period attempting to restore files that no longer exist (Crunchy Data). The same mechanism can affect a restore from a base backup if retention is shorter than the team assumed.
• Did a timeline change move the segment? Failovers and switchovers increment the timeline. A segment that exists on the old timeline may not be requested on the new one, and vice versa. The Crunchy Data article notes that time synchronization across cluster members is “critically important for correct cluster operation” (Crunchy Data), which is relevant because timeline confusion often follows clock or coordination problems.
• Was the archive command returning success when it should have failed? The PostgreSQL documentation is explicit: “It is important that the archive command return zero exit status if and only if it succeeds” (PostgreSQL 18, Continuous Archiving and PITR). An archive command that returns zero on partial writes can produce a gap that looks like a retention problem.

These are questions for the postmortem, not diagnoses. The runbook should record which questions were asked and what evidence was collected, not which answer was assumed.

What not to put in the runbook

Two things are worth leaving out.

First, do not record a specific recovery time objective, recovery point objective, or acceptable data-loss window as if it were a property of the system. Those are decisions the team makes, and they belong in a separate decision record that the runbook references. A runbook that hard-codes “acceptable loss is one hour” will be wrong the first time a different service uses it.

Second, do not record pg_resetwal as a recovery step. The Crunchy Data article advises against running it casually and notes that it may lose data and cause further damage (Crunchy Data). If the team has a documented, reviewed procedure for using it as a last resort, that procedure belongs in its own document with its own approvals, not in the PITR runbook where it might be reached for under pressure.

Similarly, the runbook should record that an OS-level copy of the data directory is made before any repair attempt. The Crunchy Data article recommends this explicitly: “make a copy of the DB’s data directory using your favorite OS-level snapshot utility” before attempting repairs (Crunchy Data). That step belongs in the runbook as a precondition, not as an optional note.

A labeled template

The following is an illustrative template, not a record of a real incident. The values are hypothetical and are marked as such. Teams should replace the bracketed fields with their own conventions.

## Restore record: [hypothetical example]

### Intent
- Target type: timestamp
- Target value: 2026-03-14 18:00:00 UTC
- Target action: promote
- Base backup: [backup label], repository [repo1]
- Requester: [role], reason: [one sentence]

### Failure
- Missing segment: 00000001000000A900000065
- Timeline: 1
- Verbatim error: [exact line from PostgreSQL log]
- restore_command: pgbackrest --stanza=[name] archive-get %f "%p"
- Observed at: 2026-03-14 18:42 UTC on [host]

### Archive lookup
- Repositories queried, in priority order: repo1, repo2
- Result from repo1: [found / not found / error text]
- Result from repo2: [found / not found / error text]
- Asynchronous archive-get: [enabled / disabled]
- archive-missing-retry: [enabled / disabled]
- Spool path cleared before restore: [yes / no]
- Timelines checked: [list]

### Decision
- Branch chosen: move target earlier
- New target: 2026-03-14 13:55:00 UTC
- Decision owner: [role]
- Trade-off accepted: writes between 13:55 and 18:00 UTC on affected tables are not present; whether any were committed is not known from the archive alone
- Decided at: 2026-03-14 19:10 UTC

### Verification
- Recovery reached: 2026-03-14 13:54:58 UTC (reported)
- Cluster accepting connections: [yes / no]
- Row counts compared: [table, expected, actual]
- Highest sequence value: [value]
- Remains uncertain: [list]

### Follow-up questions for postmortem
- Did archiving fail before the gap? Evidence: [link to pg_stat_archiver output]
- Did retention expire the segment? Evidence: [retention config, backup set age]
- Did a timeline change move the segment? Evidence: [timeline history]
- Did the archive command return success incorrectly? Evidence: [archive command, logs]

The template is deliberately longer than most teams will fill in during an incident. The point is not to complete every field under pressure; it is to have a place to put the fields that matter, so that the next reader can tell what was known, what was decided, and what was left uncertain.

Why this belongs in the runbook, not in the postmortem

The postmortem is written after the incident, by people who may not have been on call. The runbook is written before, by the team that will be on call. The runbook’s job is to preserve the decision context at the moment of decision, when the trade-off is still visible. The postmortem’s job is to ask why the gap existed and what would prevent it next time.

For a lean team, the runbook is also the onboarding document. A new engineer reading the restore record learns what the team considers a complete record: target, failure, lookup, decision, verification, follow-up. That is more useful than a paragraph of prose about the importance of backups.

If the team does not yet have a recovery checklist, the companion piece on writing the recovery checklist before you need it covers the broader structure. This article is the narrow case: what to record when the restore stops on a missing segment, and why the record is the deliverable.

FAQ

Is a missing WAL segment always a data-loss event?

No. It is a failure to reach the requested recovery target. The team may be able to retry the archive fetch, move the target earlier, or rebuild from a newer base backup. Whether data was lost depends on which branch is chosen and what the requester needed. The runbook should record the branch and the trade-off, not assume the outcome.

Should the runbook include the exact pgBackRest restore command?

Yes, if the team uses pgBackRest, because the command is part of the reproducible record. The pgBackRest command reference documents the restore options, including target and target-action, and the user guide describes the restore process. Recording the exact invocation lets the next reader reproduce the attempt or see what was different.

What if the team uses restic instead of pgBackRest for base backups?

restic can store base backup files, but it does not provide a WAL archive-get interface for PostgreSQL. The WAL archive still needs a mechanism PostgreSQL can call through restore_command or archive_library. The runbook should record which tool handles which part: restic for base backup storage, and the archive mechanism for WAL. The restic documentation index describes repository and restore operations but does not describe a PITR archive interface, so the runbook should not assume one.

How long should the restore record be kept?

At least as long as the postmortem and any follow-up actions are open, and ideally as long as the backup set it refers to is retained. A restore record that outlives its backup set is still useful for the postmortem, but it cannot be used to reproduce the restore.

Does this apply to replicas as well as primary restores?

Yes. The Crunchy Data article describes the replica case in detail: a replica that has fallen behind may request a segment the primary has already recycled, and the same decision branches apply. The runbook should record whether the restore was for a primary, a replica, or a test environment, because the acceptable trade-off differs.

The artwork is blunt: NOT FRIENDLY stacked above DO NOT TOUCH, set in the visual grammar of a hazard placard. A yellow caution triangle sits in the composition, and inside or beside it a hand is crossed out. The lettering is warning-sign lettering, not a slogan in a friendly rounded font. The whole thing reads less like a joke you wear and more like a label you post.

That distinction is the argument. Most novelty shirts make a claim about the wearer’s personality and leave it there. This one borrows the one visual system we already trust to stop behavior: industrial caution signage. A caution triangle is not decorative. It exists because someone decided that a specific hazard needed a specific response, and that the response had to work without a conversation. The crossed-out hand is the response. Do not touch. The shirt is not describing a mood. It is issuing an instruction.

We keep coming back to this because it is a useful object for the kind of teams we write for. A warning sign is a boundary that has been made legible to strangers. That is a harder engineering problem than it looks.

What the sign actually commits to

Read the three elements separately. The caution triangle is the category marker: something here warrants care. The crossed-out hand is the prohibited action: contact. The words are the plain-language restatement, because signage that relies on symbol alone fails the first time someone is distracted. Redundancy is the design principle. Symbol, prohibition, text — three channels carrying one message.

That is a recognizable pattern to anyone who has written a runbook. You do not write a step once and hope. You write the command, you note the expected output, and you name the failure mode if the output differs. The shirt does the same thing in fabric: it states the condition (not friendly), the rule (do not touch), and the visual enforcement (crossed-out hand). It is over-specified on purpose. Over-specification is what makes a sign survive contact with people who are not paying attention.

Our interpretation, not a historical claim: the humor comes from applying that machinery to a social situation. A hazard placard is impersonal by design. It does not negotiate. Wearing one turns a personal preference into public infrastructure, and the joke is the mismatch between the register and the subject. We are not asserting where the phrase came from or what any platform promotes. We are reading the object in front of us.

The difference between a slogan and a usable promise

This is where the shirt earns its place in a column about operational habits. A slogan is a sentence that sounds good and commits to nothing. A usable promise names a condition, a boundary, and what happens at the boundary. Not friendly, do not touch is a usable promise in the narrow sense: it tells you exactly what to expect and exactly what not to do. You may not like the promise. But you cannot misread it.

We have watched plenty of internal documentation fail the same test. A page that says “be careful with the production database” is a slogan. A page that says “writes to this cluster require a second approver; the break-glass role is logged and reviewed weekly” is a usable promise. The first makes everyone feel responsible and no one accountable. The second tells you the condition, the boundary, and the consequence. The shirt, whatever else it is, is on the second side of that line.

There is a cost to that clarity, and it is worth naming. A sign that says do not touch forecloses the friendly interpretation. It trades warmth for legibility. Some readers will find that refreshing; some will find it rude; both reactions are evidence that the sign worked. Ambiguous messages get ignored politely. Explicit ones get a reaction. If you have ever written an on-call policy that everyone nodded at and nobody followed, you know which kind you actually needed.

Why the crossed-out hand does the heavy lifting

The words alone could be read as self-deprecation — I’m not friendly, sorry. The crossed-out hand changes the grammar. It is not an apology; it is a prohibition. The hand is the thing being stopped, and stopping is an action the sign performs on the viewer. That shift from description to instruction is the whole design. Remove the symbol and you have a confession. Keep it and you have a boundary.

We think that is why the object holds up under a second look. The caution triangle and the crossed-out hand are doing different jobs: one classifies, one prohibits. The text repeats both in plain language. Three elements, no wasted ink. It is the visual equivalent of a well-scoped alert — one that fires on a condition you can act on, not on a vibe.

If you want the object itself, the listing is here: Not Friendly Do Not Touch warning-sign shirt. The design is the argument; the garment is just the medium.

What we’d do differently

We would steal the structure, not the tone. The next time we write a boundary into a runbook or an access policy, we would check it against the three-channel test: does it classify the condition, name the prohibited or required action, and restate both in plain language a distracted reader can follow? If a policy needs a meeting to interpret, it is a slogan wearing a placard. The shirt is a reminder that the clearest boundaries are the ones that survive being read by someone who is not in the mood to be friendly.

Editorial note: This publication and Quirky Cotton Collective share an owner.

Why a lottery beats a checklist

A restore checklist tends to test the backup you already know is easy. The snapshot you took last night, the one whose restore path you walked through during onboarding, the one on the host whose credentials are still in your shell history. That is not the backup an incident will hand you. An incident hands you whatever exists in the repository at the moment you need it, including the snapshot from a host you decommissioned two quarters ago and the one whose encryption key was rotated without a note.

A weekly restore lottery replaces the hand-picked backup with a random draw from the repository’s own listing. The randomness is the point: it removes the selection bias that makes drills feel reassuring while leaving the real recovery path untested. The hard timeout is the second half of the design. A restore that exits zero after ninety minutes against a thirty-minute budget is a failed drill, not a pass, because the question is not whether the tool works but whether your team can recover within a window it can actually tolerate.

This is a rehearsal of the recovery decision, not a test of the backup vendor. Treat it that way and the drill stays small enough to survive a busy week.

Define the two random inputs before the drill

Two things get decided before anyone touches a terminal: the snapshot and the timeout. Both are recorded in the drill log before the restore starts. Neither is adjusted mid-drill.

The snapshot is drawn from the repository listing, not from memory or a spreadsheet. For restic, restic snapshots lists every snapshot with its ID, timestamp, host, tags, and directory, and the listing can be filtered by --path, by --host, or by combining filters, and grouped with --group-by on host, paths, or tags. Draw from that output. If you keep a spreadsheet of “known good” snapshots, you are not running a lottery; you are running a checklist with extra steps.

For pgBackRest, the equivalent is pgbackrest info against the stanza. The restore command defaults to selecting the latest backup from the first repository where backups exist, and --set accepts latest or a specific backup name. The lottery should draw a specific set name, not latest, because latest is the backup you would have picked anyway.

The timeout is a wall-clock budget agreed in advance, derived from what the team can tolerate during an incident rather than from how long the restore usually takes. It does not change between weeks, because changing it destroys the comparison. If the budget turns out to be wrong, change it deliberately between drills and note the change in the log.

Example: drawing the snapshot

# restic: list snapshots, then pick one at random
restic -r s3:s3.amazonaws.com/example-restic snapshots --json \
  | jq -r '.[].short_id' \
  | shuf -n 1

Record the chosen ID and its timestamp in the drill log before starting the restore. The same pattern works for pgBackRest: parse pgbackrest info --output=json, extract the backup labels for the stanza, and draw one. The tool’s own listing is the source of truth for what exists; a stale inventory is not.

Set the timeout as a hard stop

Start a timer when the restore command starts. Stop the drill when the timer expires, regardless of how close the restore looks to finishing. A timeout is a finding, not a failure to hide. The drill log records it as one of three outcomes: completed within timeout, completed but over timeout, or timed out. The third outcome is the most informative, because it tells you the recovery path has a step that does not fit inside the window you would actually have during an incident.

Do not extend the timeout to see whether the restore would have finished. That converts a bounded operational fact into an unbounded curiosity, and it teaches the team that the budget is negotiable. If the restore is still running when the timer expires, stop it, record where it was, and clean up the target.

Run the restore into an isolated target

The drill restores into a target that is not production and not reachable from production. For restic, restic restore <snapshotID> --target /srv/restore-drill/<date> extracts the snapshot to a directory; the special snapshot ID latest exists but the lottery should not use it. For pgBackRest, the restore command writes into the PostgreSQL data and tablespace paths, which by default are expected to be present but empty; the --delta option performs a delta restore using checksums, and --target-action controls what the cluster does when a recovery target is reached, with pause as the default.

Isolation matters for two reasons. First, a restore that writes into a live cluster’s data directory is not a drill; it is an incident. Second, the isolated target is where you can afford to let the restore run to its timeout without worrying about what it is overwriting.

Verify restored content, not just exit status

An exit status of zero means the restore command completed, not that the restored data is correct. restic’s restore command has a --verify flag described as “verify restored files content”; per the restic maintainers, it enables an additional check after the restore has been done, which goes over all files, reads them again, and verifies that the correct data is read. It is mutually exclusive with --dry-run. It is not a full repository integrity check, and it does not concern itself with additional files in the target tree; it checks the files the snapshot lists.

For PostgreSQL, verification is a different shape. The restore is only meaningful if the cluster can start and replay WAL to a consistent state. PostgreSQL’s continuous archiving documentation is explicit that to recover successfully using continuous archiving, you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup. A restore that produces a data directory but cannot find the WAL segment it needs is a failed drill even if the restore command exited zero. The drill should include starting the restored cluster and confirming it reaches consistency, or at minimum confirming that the WAL archive contains the segments the backup’s history file names.

This is also where the drill catches archiving problems that monitoring missed. PostgreSQL’s documentation advises monitoring the archiving process to ensure it is working as intended; a restore lottery that regularly fails to find WAL is a stronger signal than a dashboard that says archiving is green.

Capture three outcomes and one change

The drill log is short by design. A long template invites skipping; a short one survives busy weeks. Each entry records:

• What was drawn: the snapshot ID, its timestamp, the host or stanza, and the repository.
• The budget: the wall-clock timeout agreed in advance.
• The outcome: completed within timeout, completed but over timeout, or timed out.
• Where time was lost, if any: credential retrieval, network path, target provisioning, WAL retrieval, verification.
• One concrete change for next week.

The “where time was lost” field is the one that earns its keep. A restore that times out during credential retrieval is a different problem from one that times out during WAL replay, and the fix for each is different. Recording the step, not just the total, lets next week’s drill target the slow part.

The “one concrete change” field keeps the drill from becoming a ritual. If the same step is slow three weeks running, the change should be to that step, not to the drill.

Keep the note blameless and the cadence honest

The drill log is a blameless artifact. It records what was tested, what the budget was, where it stalled, and what changes next. It does not record who ran the drill badly or whose credential was missing. The Google SRE Workbook’s chapter on postmortem culture makes the case that a truly blameless postmortem culture results in more reliable systems, and recommends starting small with a very basic procedure and then reflecting and tuning it. The restore lottery log is that basic procedure applied to recovery rehearsal: short, factual, and acted upon.

Two habits keep the cadence honest. First, rotate who runs the drill, so recovery knowledge is not concentrated in one person. The lottery is also a knowledge-distribution mechanism; the person who draws the snapshot and watches the timer learns where the recovery path actually goes. Second, do not skip the drill during busy weeks. The busy weeks are the ones the drill is for, because they are the weeks when the team’s recovery capacity is most likely to be degraded and the drill is the only signal that it has degraded.

If a week is genuinely too busy to run a full restore, run a smaller version: draw the snapshot, start the timer, and stop at the first step that requires a credential or a network path you do not have. That is still a finding, and it is still cheaper than discovering the gap during an incident.

What the lottery is not

The restore lottery is one rehearsal among several. It does not replace periodic full restore tests against production-shaped infrastructure, repository integrity checks, or monitoring of the backup and archiving pipelines. It does not prove the backup tool is correct; it proves the team can execute a recovery decision within a bounded window on a backup it did not choose. Those are different claims, and conflating them is how drills become theater.

It also does not require a dedicated SRE function. A two-person team can run it in under an hour if the target is pre-provisioned and the credentials are already in the drill runner’s hands. The hard timeout is what keeps it from expanding to fill the available time.

FAQ

Should the timeout be the same every week?

Yes, until you deliberately change it. The timeout is the comparison point across weeks; changing it mid-drill or week-to-week destroys the signal. If the budget turns out to be wrong, change it between drills and record the change.

What if the random snapshot is from a host we no longer run?

Restore it anyway. That is exactly the kind of backup an incident might hand you, and the drill is the cheapest place to discover that the restore path for a decommissioned host no longer works. If the snapshot is genuinely out of scope, note why in the log and draw again, but do not make a habit of redrawing.

Does restic’s --verify prove the backup is good?

No. It verifies that the restored files match the snapshot’s recorded content, after the restore has completed. It does not check the repository’s integrity as a whole, and it does not concern itself with files outside the snapshot. Pair it with a repository check if you want integrity assurance; the lottery is about recovery execution, not repository health.

How does this interact with pgBackRest’s --delta?

--delta performs a delta restore using checksums, which is useful when the target already contains a previous restore. For a drill, a clean target is simpler and produces a cleaner timing signal; use --delta when you are rehearsing the incremental path specifically, and note in the log that you did.

What if the restore finishes early?

Record the elapsed time and stop. Finishing early is a good outcome, not an invitation to add scope. The next week’s draw is the next test.

Related: Write the Recovery Checklist Before You Need It covers the pre-incident checklist that the lottery log feeds into.

There is a period at the end of the word on this shirt. That is the first thing we noticed, and it is the thing that keeps the design from collapsing into a mood. The native garment preview shows very small, centered lowercase meh. lettering on a dark shirt. No animal illustration is present. The scale is almost apologetic; the punctuation is not.

We keep coming back to that period because it changes what the garment is doing. A bare meh is a sound. It is the noise a person makes when a question has been answered adequately and no further energy is available. With a period, it becomes a sentence. A sentence has a speaker, a subject, and a boundary. The shirt is not advertising a personality type. It is issuing a small, complete verdict and then declining to elaborate.

That distinction matters more than it sounds. In operational work, we spend a lot of time around messages that are technically complete but socially unfinished. A monitoring alert fires, someone acknowledges it, and the channel goes quiet. The silence is not the same as resolution. The same is true of a one-word reply in a code review: it can read as a considered judgment or as a door closing, depending on punctuation, context, and what the reader needed. The artwork here picks the period, and the period is the whole argument.

What the artwork actually commits to

We should be precise about what is printed and what we are reading into it. The visual observation is narrow: very small, centered lowercase lettering on a dark shirt, no animal illustration. The retail title uses words like funny, sarcastic, introvert, and minimalist. Those are merchandising categories, not printed text. We are not treating them as evidence of what the shirt says or what wearing it feels like.

What the design commits to, visually, is understatement. The word is small enough that a reader has to be close to parse it. That is a different social contract from a chest-sized slogan. A large slogan asks the room to agree before anyone has spoken. A small one rewards the person who leans in. It also gives the wearer an exit: if the joke does not land, it was never loud enough to require a defense.

We read the period as the design’s only assertive mark. Everything else is deferential — lowercase, centered, modest in scale. The period is where the sentence lands. It says the thought is finished, not that the wearer is finished with the conversation. That is a useful distinction for anyone who has watched a terse message get read as hostility when the sender intended only closure.

The difference between a slogan and a usable promise

Most novelty shirts are slogans. They make a claim about the wearer’s temperament and invite the room to sort people into camps. This one is closer to a status line. It reports a state without asking for sympathy or agreement. That is a harder thing to print, because it depends on restraint rather than volume.

We think about this in terms of team communication. A slogan is a broadcast: it assumes an audience and performs for it. A status line is a narrowcast: it tells the people who are paying attention what the current state is. The small meh. reads like the second kind. It does not promise that the wearer is funny, deep, or introverted. It promises that the wearer has reached a conclusion and is not going to pad it.

That is also why the design resists the usual novelty-shirt escalation. There is no second joke, no arrow, no animal reacting to the word. The absence of an illustration is part of the composition. A drawing would give the viewer something to interpret; the bare word with its period gives them only the sentence. We are not claiming the designer intended a treatise on boundaries. We are claiming the object, as observed, supports that reading better than it supports a laugh-out-loud gag.

Why the small scale is the point, not a limitation

There is a temptation to treat small typography as a compromise — something chosen because a bigger print would cost more or crowd the garment. We do not read it that way. The scale is doing semantic work. A word that means diminished enthusiasm should not be shouted. If the lettering were large, the shirt would be performing exhaustion rather than reporting it. The small centered mark keeps the report credible.

This is the same reason we prefer terse runbook entries to dramatic ones. A runbook line that says restore failed at 03:14; WAL gap from 02:58 is more useful than a paragraph about how stressful the night was. The paragraph may be true, but it is not the operational content. The shirt’s design makes a similar editorial choice: it gives you the word, the period, and nothing else. Whether that is funny is a matter of taste. Whether it is legible as a complete thought is not.

We also note what the design avoids. It does not use a distressed texture, a faux-vintage banner, or a second line of smaller text underneath. Those choices would push it toward costume. The plain small word on a dark ground reads as a garment first and a joke second, which is the more durable arrangement.

Reading the shirt without overclaiming it

We want to be careful here. Nothing about a printed word demonstrates a software practice, a security posture, or an engineering result. The shirt is not evidence that its wearer is good at incident response, and it is not a diagnostic tool for team morale. It is a piece of clothing with a sentence on it. Our interest is in the sentence’s construction, not in what it proves about the person wearing it.

That said, the construction is worth noticing because it is the same construction we use when we write a decision record. A decision record is not persuasive by volume. It states the decision, the date, and the tradeoff, and then it stops. The period at the end of meh. is doing the same kind of work: it marks the end of the reasoning without pretending the reasoning was elaborate. If you have ever written a one-line postmortem action item and watched it get ignored because it was too short to seem serious, you know the tension the design is playing with.

The shirt is available as a standard t-shirt through the listing: Meh Funny Sarcastic Introvert Minimalist Saying. We are describing the observed artwork, not a wearing experience. We have not tested the fabric, fit, or print durability, and nothing in the preview establishes those.

What we’d do differently

If we were writing the retail copy, we would drop the word funny from the title. The design’s strength is that it does not ask to be laughed at; it asks to be read. Calling it funny sets an expectation the artwork deliberately undercuts. We would also resist adding an animal or a second line in a future variant. The restraint is the product. A louder version would be a different shirt with a different argument, and probably a weaker one.

For our own internal writing, the lesson we are taking is about the period. When we send a short message that closes a thread, we should make the closure explicit rather than relying on brevity to carry it. A terse reply is not the same as a rude one, but the reader cannot always tell. The shirt gets to rely on a printed period. In a chat channel, we have to supply the equivalent ourselves.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner. This column is an owner-authorized reader-interest piece, not an independent review.

A backup job that exits zero every night answers one question: did the upload complete. It does not answer whether the repository is internally consistent, whether a restore would succeed, or whether the snapshot contains the data you believe it does. For a team of two to fifteen engineers who are their own on-call rotation, that gap is the whole risk. The nightly cron is green; the restore path has never been exercised.

This article is about closing that gap without decrypting production data and without standing up a second production environment. The premise needs one correction up front: you cannot validate an encrypted backup without ever decrypting anything. restic’s check and restore both require the repository key. What you can do is keep every validation step off production paths, off production credentials, and off production data — and separate the work into three checks that can be scheduled independently.

Three questions, three checks

“Is the backup valid?” is not one question. It decomposes into:

1. Integrity. Is the repository internally consistent, and are the stored data files unmodified?
2. Restore-path proof. Can we actually restore a snapshot to a target we control?
3. Coverage. Does the snapshot contain the hosts, paths, and data we think it does?

Each check has a different cost and a different failure mode. Splitting them lets a small team run the cheap ones often and the expensive ones on a rotation, rather than trying to do everything in one long maintenance window nobody wants to own.

Listing is not validating

restic’s snapshots command lists snapshot metadata: ID, date, host, tags, directory, and — for snapshots created with restic 0.17.0 or later — size. The ls command lists the files inside a specific snapshot, and the special snapshot ID latest selects the most recent one. Both are useful for coverage checks. Neither reads the repository’s data files.

A green backup cron plus a successful restic snapshots is the weakest possible evidence. It proves the repository index is readable. It does not prove the packs behind that index are intact.

Layer one: integrity

restic’s own documentation is explicit about the default: “By default, check command does not check that repository data files are unmodified.” The check command verifies the repository’s structure — indexes, snapshots, trees, and blobs — but reading the actual data files requires an additional flag.

Two flags change that:

• --read-data reads all repository data files.
• --read-data-subset=n/t divides all repository data files into t roughly equal groups and checks only group n.

The subset flag is the affordance that makes integrity checking practical for a lean team. A full --read-data pass on a large repository is a long-running job that competes with backup windows and on-call attention. Spreading it across t scheduled invocations amortizes the cost: run --read-data-subset=1/5 this week, 2/5 next week, and so on, and every data file is read once per cycle. The restic documentation shows exactly this pattern across five invocations.

Two operational notes from the same documentation. First, restic advises running restic check after prune, so that damage to internal data structures surfaces immediately rather than at the next drill. Second, the check command is presented as the regular mechanism for confirming that backup data is consistent and its integrity is unharmed — not a one-time setup step.

What this layer does not prove: that a restore will succeed end to end, that the snapshot contains the right data, or that the repository is tamper-evident against an attacker with write access. It proves the stored bytes are readable and consistent with the repository’s own structures. That is a real property, and it is not the same as recoverability.

Layer two: restore-path proof

The only check that proves the restore path is a restore. The discipline is in where it lands.

Restore to an isolated scratch target with its own credentials — a separate host, a separate bucket, a separate filesystem that shares no identity with production. The point of the drill is to exercise the restore path, not to test your ability to be careful under pressure. If the drill target can reach production paths, the drill is one typo away from an incident.

A drill script shape, clearly hypothetical and meant to be adapted rather than copied:

# Runs on an isolated drill host with its own repository credentials.
# Never point RESTIC_REPOSITORY at a production path.
export RESTIC_REPOSITORY="s3:s3.example-region.amazonaws.com/drill-restic"
export RESTIC_PASSWORD_FILE="/etc/restic/drill-password"

SNAPSHOT_ID="$(restic snapshots --host app-01 --json | jq -r '.[-1].short_id')"
TARGET="/srv/drill/restore-${SNAPSHOT_ID}"

mkdir -p "${TARGET}"
restic restore "${SNAPSHOT_ID}" --target "${TARGET}"

# Assertions: the restore must produce the files you expect.
test -f "${TARGET}/srv/app/config/production.yaml"
test -d "${TARGET}/srv/app/uploads"

# Record the result for the runbook.
echo "drill ok snapshot=${SNAPSHOT_ID} target=${TARGET} at=$(date -u +%FT%TZ)"

The assertions matter more than the restore. A restore that completes but produces an empty directory is a failed drill. Pick a small set of files and directories whose presence is non-negotiable — the application config, the upload directory, the database dump if you take one — and assert on them explicitly.

One cost note from restic’s documentation: the copy command must download and upload the entire snapshot because source and destination repositories use different encryption keys. If your drill strategy involves copying snapshots to a separate drill repository, budget for that bandwidth. Restoring directly from the offsite repository to the drill host avoids the copy entirely and is usually the better shape for a small team.

Layer three: coverage

Coverage is the cheapest check and the one most often skipped. It answers: does the snapshot contain the hosts and paths we believe it does?

restic’s snapshots command filters by --path and --host, and combines filters. A coverage check is a short script that asserts, for each host in your inventory, that a snapshot exists within the expected window and includes the expected paths. Run it daily. It catches the failure mode where a backup job silently stops including a directory after a config change — a failure that integrity checks and restore drills will not catch if the drill happens to target a snapshot from before the change.

Coverage is also where you catch the inverse: a snapshot that includes more than you intended. If a path was added to the backup set by accident, coverage assertions on the expected set will not flag it, but a periodic review of restic ls --long latest output against the intended inventory will.

The PostgreSQL case: a file restore is half the story

For teams running PostgreSQL with continuous archiving, a file-level restore drill proves the base backup is readable. It does not prove point-in-time recovery works. The PostgreSQL documentation is direct: “To recover successfully using continuous archiving … you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup.” The unit of recovery is the base backup plus the WAL chain from that backup forward, not the base backup alone.

That means the PostgreSQL drill has its own shape: restore the base backup to an isolated target, configure recovery to replay WAL to a chosen target time, start the server, and assert on the state of a known table. The PostgreSQL documentation also advises setting up and testing the WAL archiving procedure before taking the first base backup — the drill is not an afterthought to the archiving setup, it is part of it.

Two related cautions from the same documentation. pg_dump and pg_dumpall do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution; a logical dump drill does not substitute for a PITR drill. And WAL archiving does not capture changes to postgresql.conf, pg_hba.conf, or pg_ident.conf, so those files need to be in your regular file-system backup set and covered by the coverage check.

pgBackRest’s verify command is a useful complement, not a replacement. It checks database and archive files and reports counts of missing, checksum-invalid, and size-invalid files per backup — the output shape is visible in the project’s issue tracker. But the --set option to scope verification to a single backup is marked “FOR INTERNAL USE ONLY. DO NOT USE IN PRODUCTION” in its help text, per the same issue. Treat verify as an integrity layer and keep the PITR rehearsal as a separate drill.

What about age-encrypted files?

If your offsite layer wraps artifacts in age, the format gives you a structural check that does not require the identity. An age file is a textual header carrying a per-file 128-bit symmetric file key, followed by a binary payload encrypted with that key. The header ends with an HMAC-SHA-256 over the header itself, keyed from the file key. The payload is split into 64 KiB chunks, each encrypted with ChaCha20-Poly1305, and streaming decryption must signal an error if the end of file is reached without a successfully decrypted final chunk.

What that buys you: a truncated or bit-flipped age file will fail to decrypt, and the header MAC will fail if the header was modified. What it does not buy you: a repository-level tamper-evidence guarantee. The age specification describes the file format, not a threat model for an offsite store where an attacker has write access. If tamper-evidence matters, that is a separate control — object versioning, immutability, or an append-only repository configuration — not a property of the encryption format.

What object storage can and cannot tell you

S3-compatible stores expose checksums over object bytes at rest. Amazon S3 documents a Compute checksum operation in S3 Batch Operations that calculates checksums for a list of objects without restoring or downloading them, and produces an integrity report. That is a real and useful control: it tells you the bytes S3 holds match the bytes S3 recorded at upload.

It does not tell you the restic repository is logically consistent. Object-level checksums verify storage integrity; restic’s check verifies repository structure. They are different properties, and a team that runs only one of them has a gap. Run both, and label them correctly in the runbook so the next on-call engineer knows which failure each one catches.

Make the drill fail on purpose

A drill that has never failed is a drill whose assertions are too weak. Periodically, in a non-production test repository, corrupt or truncate an artifact deliberately and confirm the drill catches it. restic’s documentation includes exactly this demonstration: appending bytes to an index file causes restic check to report a ciphertext verification failure, and a restore of the affected snapshot fails with the same error. Reproduce that in your own test repository once a quarter. If the drill passes on a deliberately corrupted repository, the drill is not testing what you think it is.

Record the evidence, not the Slack message

The artifact of a drill is inherited evidence. Record, for each run: date, snapshot ID, subset index (for integrity runs), restore target, assertions executed, and outcome. Put it in the same runbook or decision record the on-call rotation reads. A drill result that lives only in a Slack thread is lost at the next rotation handoff.

This is the same discipline described in Write the Recovery Checklist Before You Need It: the checklist is written when the pressure is low so that it can be followed when the pressure is high. The drill record is the checklist’s evidence trail.

A minimal drill checklist for a small team

Adapt the cadence to your repository size and your tolerance for long-running jobs. The structure matters more than the specific numbers.

• Daily: coverage check. Assert that each host in inventory has a snapshot within the expected window and includes the expected paths. Cheap, fast, catches silent scope changes.
• Weekly: restic check without --read-data, plus restic check after any prune. Catches structural damage early.
• Rotating: restic check --read-data-subset=n/t, one subset per scheduled run, so every data file is read once per cycle. Choose t so the cycle fits your risk tolerance and the per-run cost fits your team’s attention.
• Rotating: restore drill to an isolated scratch target with its own credentials. Assert on a small set of non-negotiable files. Record the result.
• Rotating, PostgreSQL shops: PITR rehearsal — restore base backup, replay WAL to a target time, start the server, assert on known state. This is a separate drill from the file restore.
• Quarterly: deliberate-corruption test in a non-production repository. Confirm the drill catches what it is supposed to catch.

The nightly backup cron stays. It is the thing that produces the artifacts the drill validates. It is not the validation.

FAQ

Can I validate an encrypted backup without decrypting it at all?
Not meaningfully. restic’s check and restore both require the repository key. Object-storage checksums and age header structure can be inspected without the key, but they verify storage integrity and file format, not recoverability. The realistic goal is to keep decryption off production paths and production credentials, not to avoid it entirely.

Does restic check --read-data guarantee the backup is recoverable?
No. It reads the data files and confirms they are consistent with the repository’s structures. It does not prove a restore will produce the files you expect, and it does not prove the snapshot contains the right data. That is why the restore-path and coverage layers exist separately.

Is pgBackRest verify a substitute for a PITR drill?
No. verify checks database and archive files and reports missing, checksum-invalid, and size-invalid counts. It does not replay WAL to a target time and start a server. The PITR rehearsal is a separate exercise.

How often should the restore drill run?
There is no sourced answer. The tradeoff is between the cost of the drill and the age of the evidence you are willing to carry into an incident. Pick a cadence you will actually sustain, record it, and revisit it when the repository grows or the team changes.

What if the drill target is expensive to stand up?
The drill target does not need to be a full production replica. It needs to be isolated, to have its own credentials, and to be able to receive a restore and run assertions. A small instance with enough disk for the snapshot you are testing is sufficient for the restore-path layer. The PITR layer needs a PostgreSQL instance, but it can be a single-node scratch instance with no production connectivity.

The native garment preview shows a dark shirt with large, bold, white lowercase lettering: meh. The period is part of the word, not a separate mark. There is no animal illustration, no secondary slogan, no gradient. The type is heavy enough to read across a room, and the word it carries is a shrug.

That mismatch is the whole design. A shirt that says meh in 72-point weight is not expressing indifference. It is performing indifference at volume. The typography argues with the word.

We keep a small decision log for our on-call rotation. One recurring entry is the difference between a message that states a position and a message that performs one. A runbook line that says “investigate the alert” is a performance of diligence. A runbook line that says “check pg_stat_archiver.last_failed_time; if it is within 10 minutes, page the database owner” is a position. The first sounds responsible. The second is responsible.

The meh shirt sits on the other side of that distinction, and that is what makes it useful to look at. It is a slogan, not a promise. It does not claim the wearer is calm, or busy, or above it all. It just puts a large, declarative meh. on the chest and lets the reader supply the context. In a work culture that rewards visible enthusiasm and visible urgency, a shirt that refuses both is a small, legible boundary marker. It says: I am here, I am not performing, and the period is final.

We noticed the same pattern in incident communication. The most useful status updates we send are not the ones that sound most alarmed. They are the ones that name the current state and the next checkpoint in plain language: “WAL archive gap confirmed; restore drill scheduled for 14:00; no customer impact yet.” That is not meh in tone, but it shares the shirt’s structural honesty. It refuses to inflate. It refuses to understate. It picks a word and commits to it.

The shirt’s word is a refusal, not a diagnosis. It does not tell you what the wearer thinks of the meeting, the deploy, or the request. It tells you that a verdict has been reached and will not be elaborated. That is a specific social move, and the bold lowercase type makes it legible without making it aggressive. The period does the work of closing the sentence before anyone can append a question mark.

If you want the object itself, the listing is here: Meh Funny Sarcastic Saying Introvert Dry Humor. We are not reviewing the fabric, the fit, or the print durability. We are reading the artwork, which is the only thing the preview actually shows.

Editorial note: This publication and Quirky Cotton Collective share an owner.

What we’d do differently

We would resist the urge to turn a shirt into a lesson about incident response. The connection is real but narrow: both the shirt and a good status update depend on choosing a word and refusing to decorate it. We would keep the observation at that level and not pretend the garment demonstrates anything about reliability engineering. The artwork is a visual argument about tone. It is not evidence about systems.

We would also read the period more carefully next time. It is small, but it is the part of the design that turns a sound into a sentence.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The shirt shows a white curved ramp descending from left to right. Above it, in white capitals: I HAD POTENTIAL. At the upper left ledge sits a dashed empty circle. At the lower right, on the flat runout, rests a solid white ball. The ramp is curved, not a straight diagonal. That curve is the whole argument.

We keep coming back to this design because it is not a slogan with a picture attached. It is a diagram with a sentence attached, and the two are in tension. The sentence is past tense and first person: I had potential. The drawing is present tense and third person: a ball, a ramp, a dashed outline where something used to be. The joke only works if you read both. The dashed circle is the potential — an outline, a position, a shape that could have been occupied. The solid ball is what actually happened. It rolled down the curve and came to rest on the flat. Nothing exploded. Nothing was destroyed. The ball is fine. It is just no longer at the top.

That is a more precise joke than the phrase alone suggests. A straight diagonal would have made the same point faster and worse. A straight ramp implies a single slope, a clean acceleration, a predictable outcome. The curve implies something else: the path changed as it went. The ball did not fall off a cliff. It followed a shape. The shape was there before the ball arrived. That is the part we find useful as an observation, not as a lesson.

We spend a lot of working hours around diagrams that describe systems rather than people. A runbook is a diagram of a procedure. A decision tree is a diagram of a choice. An escalation policy is a diagram of who gets woken up. The best of them share a property with this shirt: they show the shape of the path, not just the endpoint. A runbook that says “restore the database” is a straight diagonal. A runbook that says “check the WAL archive gap before you promote the replica, because the last drill showed a 40-second window where the primary and standby disagreed” is a curve. It costs more to write. It is more useful at 3 a.m.

The shirt is not a runbook. We are not going to pretend it demonstrates anything about backup recovery, incident response, or the reliability of a small team’s on-call rotation. It is a piece of printed cotton with a physics-adjacent visual pun on it. But the visual pun is doing real work, and it is worth naming what that work is. The dashed circle is a counterfactual. The solid ball is an observation. The gap between them is the joke. Most slogan shirts collapse that gap by asserting the punchline outright. This one leaves the gap visible and lets the reader close it.

There is a second reading, and we think it is the one that makes the shirt worth wearing rather than just worth noticing. The ball is not lost. It is resting. The ramp did not fail; it delivered the ball to a flat ledge, which is where balls go when ramps end. The sentence says had potential, but the drawing says reached equilibrium. Those are not the same claim. One is a regret. The other is a description of a system that has settled. We are not going to argue that the shirt is secretly optimistic — that would be overreading a garment — but we will note that the artwork is more ambiguous than the text. The text is a verdict. The drawing is a state. If you have ever written a postmortem that concluded “the system behaved as designed, and the design was the problem,” you have lived in that gap.

The design is also unusually restrained for a novelty shirt. White ink on a dark ground, no secondary colors, no distressed texture, no additional copy. The only punctuation is the period implied by the end of the sentence, which is not printed but felt. The dashed circle is the lightest element on the shirt and the most important. It is the only part of the drawing that is not solid, and it is the only part that refers to something that is not there. That is a deliberate choice, and it is the kind of choice we notice because we make similar ones in diagrams: the empty box, the dotted line, the placeholder that says this is where the thing would go if it existed. A dashed outline is how a diagram admits a counterfactual without asserting it.

We keep a small decision log for our own work. One of the entries we return to is a note about the difference between a promise and a description. A promise says what will happen. A description says what did happen. Most operational writing fails by confusing the two: the runbook promises a clean failover, the postmortem describes a 90-second window of split-brain, and the gap between them is where the next incident lives. The shirt is a description. It describes a ball on a ledge and a dashed circle on a ramp. It does not promise that the ball will roll back up. It does not promise that the dashed circle will be filled. It just shows you the shape and lets you decide what to do with it.

If you are the kind of reader who keeps a pager and a decision log, the shirt is available as a standard t-shirt from Quirky Cotton Collective. We are not going to tell you it will improve your on-call rotation. It is a shirt. But it is a shirt with a diagram on it, and the diagram is more careful than it needed to be. That is worth something.

What we’d do differently

If we were writing the copy for this shirt, we would resist the urge to explain the joke in the product title. The current title stacks “funny,” “physics pun,” “sarcastic,” and “minimalist” into a single line, which is four claims where the artwork makes one. The artwork is minimal. The title is not. We would let the diagram do the work and keep the retail prose closer to the observation: a curved ramp, a dashed circle, a solid ball. That is the whole object. Everything else is a search keyword wearing a hat.

We would also keep a note in our own decision log about the difference between a dashed outline and a solid one. The dashed circle is the part of the design that carries the meaning, and it is the part most likely to be lost in a thumbnail. If we were designing a diagram for a runbook, we would ask whether the empty box is doing the same work: is it showing a counterfactual, or is it just a placeholder we forgot to fill? The shirt is a reminder that the empty shape can be the point. We would write that down before the next incident, not after.

Two clocks, one row

Most postmortem timelines have a single row that reads like this: 14:07 — alert fired; on-call investigated and mitigated. That row hides two different failures. Detection asks how long it took for any signal to reach a human. Diagnosis asks how long it took for a human to hold a correct model of what was breaking. A lean team that collapses both into one timestamp cannot tell whether its next investment belongs in monitoring or in runbooks, access, and observability.

This is a writing and process problem, not a tooling problem. The templates most teams already use do not have a dedicated diagnosis field. PagerDuty’s published postmortem template, for example, asks for the time the contributing factor began, the time of the page, the time the status page was updated, the time of significant actions, and the end time, each with a link to the tool or log that shows how the timestamp was arrived at. It does not ask when someone first understood the failure. The fix is to add boundary rows to the template you have, with the same evidence discipline the template already demands.

Define four boundaries before the incident

The SRE Book is direct that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary. The same logic applies inside the document. Agree on four boundary events in advance, and write them into the template as empty rows:

1. Failure onset. When the system first departed from normal behavior, whether or not anyone could see it. This is often reconstructed from logs and is the least certain row in the table.
2. First signal. When any signal reached a human or a system that would act on it: an alert firing, a synthetic check failing, a customer report arriving. This is detection time.
3. First correct diagnosis stated aloud. When someone wrote or said a hypothesis that turned out to be the actual cause. This is diagnosis time. It is not the same as acknowledgment, and it is not the same as the first hypothesis.
4. First mitigation action. When someone changed the system to reduce impact, even if the change was temporary.

Two derived gaps fall out of these rows: signal-to-human (detection) and human-to-correct-model (diagnosis). They point at different remediation work. A long detection gap is a monitoring or coverage problem. A long diagnosis gap is a context, runbook, access, or observability problem. An action item that does not name which gap it closes is hard to prioritize six weeks later.

Use the existing template as a floor, not a ceiling

PagerDuty’s template is a good floor because it insists on traceability: every timestamp should link to the tool or log that justifies it. Keep that discipline and extend the table. A workable column set for a small team is:

The caveat column matters more than it looks. Monitoring data has its own lag, and a timeline that treats a dashboard timestamp as ground truth can understate or overstate detection. PostgreSQL’s documentation notes that each server process flushes accumulated statistics to shared memory just before going idle, but not more frequently than once per PGSTAT_MIN_INTERVAL milliseconds (one second unless altered at build time), so displayed totals lag actual activity. The same documentation notes that in the default configuration, statistics accessed within a transaction are cached until that transaction ends, so a long-running query can show static numbers. On the AWS side, CloudWatch preserves alarm history for 30 days and marks each state transition with a unique timestamp, and an alarm invokes actions only when it changes state. If your first signal came from an alarm that had been in ALARM for twenty minutes before anyone looked, the alarm history will show the transition, not the human noticing. Record both.

Three rows, not one: paged, acknowledged, understood

For a team of two to fifteen engineers who are their own on-call rotation, the most common diagnosis gap is not ignorance. It is context switching. The on-call is mid-task, acknowledges the page, finishes a sentence or a deploy step, and only then begins real diagnosis. If the timeline records only “acknowledged,” that gap is invisible and the postmortem will misattribute the delay to tooling.

Split the human side into three rows:

• Page fired. The alert or escalation event, with its tool timestamp.
• Acknowledged. The moment a human claimed the page. This is a tool log, not a memory.
• First correct diagnosis. The moment a correct hypothesis was stated in chat or on a call. This is usually reconstructed, and should be labeled as such.

The middle row is the one lean teams most often skip, and it is the one that makes the context-switch cost visible without blaming the person. Write it as a system state: “page acknowledged while a deploy was in progress,” not “on-call was slow to respond.”

Log the hypotheses that died

Diagnosis is iterative, and the final answer is rarely the first guess. A timeline that records only the correct diagnosis reads as if the team knew immediately, which makes the diagnosis gap look shorter than it was and hides the evidence that killed the wrong hypotheses. Add a short hypothesis log beneath the boundary table:

H1 (14:12): Primary overloaded by traffic spike. Killed at 14:18 — request rate flat in the load balancer metrics.
H2 (14:19): Network partition between primary and replica. Killed at 14:23 — replication connection alive, lag growing steadily.
H3 (14:26): WAL retention exhausted, replica cannot catch up. Confirmed by archive status and disk usage.

Each hypothesis carries the evidence that killed it. This is the decision-log part of the timeline, and it is what a reviewer needs to re-derive the diagnosis gap rather than trust the author’s summary. It also produces better action items: H1 and H2 were killed by metrics that existed, which suggests the diagnosis gap was not a missing-signal problem. H3 required a check that lived in a runbook nobody had opened, which suggests a different fix.

Keep the language factual

Blamelessness is a formatting constraint as much as a cultural one. The SRE Workbook’s critique of a poorly written postmortem calls out animated language and dramatic descriptions as distractions that erode psychological safety, and recommends providing verifiable data to justify the severity of a statement. The SRE Book defines a blameless postmortem as one that focuses on contributing causes without indicting any individual or team, and assumes everyone involved did the right thing with the information they had. Google’s CRE guidance goes further for shared documents: express the role of humans in terms of role rather than name.

In practice, that means the timeline says “the on-call” and “the alert,” not names. It says “the runbook did not cover WAL retention” rather than “the on-call did not know.” It says “the alert fired at 14:07 and was acknowledged at 14:11” rather than “response was slow.” The boundary rows make the gaps visible; the language keeps them reviewable.

A worked example, clearly hypothetical

Consider a PostgreSQL replica-lag incident on a small team running pgBackRest and streaming replication. The alert fires at T+0. The on-call acknowledges at T+4 while finishing a deploy. The first hypothesis is a traffic spike; it dies at T+11 when the load balancer metrics are flat. The second hypothesis is a network partition; it dies at T+17 when the replication connection is shown alive. The correct diagnosis — WAL retention exhausted on the primary, so the replica cannot catch up — lands at T+19. Mitigation begins at T+24 when a batch writer is paused.

The boundary table looks like this:

• Detection gap: T+0 (signal) minus failure onset. If onset was T−6, detection took six minutes.
• Acknowledgment gap: T+4 minus T+0. Four minutes of context switching.
• Diagnosis gap: T+19 minus T+4. Fifteen minutes of sense-making, including two dead hypotheses.
• Mitigation gap: T+24 minus T+19. Five minutes to act once the model was correct.

Now attach action items to rows. The detection gap suggests a review of the alert’s evaluation window and whether a leading indicator existed earlier. The acknowledgment gap suggests either an escalation policy change or an honest acknowledgment that a single on-call rotation will always have context-switch cost, which is a staffing tradeoff rather than a tooling bug. The diagnosis gap suggests a runbook entry for WAL retention and a dashboard panel that shows archive status next to replica lag. The mitigation gap suggests a pre-approved action — pausing the batch writer — that does not require a second pair of eyes at 2 a.m.

None of these action items is novel. What is novel is that each one names the row it closes, which makes the postmortem review meeting a prioritization exercise rather than a re-litigation of the incident.

What the timeline is for

A postmortem timeline with explicit boundaries is a decision log, not a narrative. The SRE Workbook notes that a good postmortem lets readers know what happened, how the issue was mitigated, and how users were impacted — and that the recovery section is where many drafts fall short. The boundary rows are how the recovery section gets specific. The SRE Book lists a monitoring failure, which usually implies manual incident discovery, among common postmortem triggers; a timeline that separates detection from diagnosis is how you tell whether a given incident was that kind of failure or a different one.

For a team without a dedicated reliability function, the timeline is also the only durable record of how the on-call rotation actually behaves under load. The recovery checklist you write before you need it — the one that covers restores, access, and escalation — is a different artifact, but it shares the same discipline: name the step, name the evidence, name the owner. If your team is still building that habit, the recovery checklist article is a useful companion to this one.

Review checklist

Before the postmortem meeting, check the timeline against these questions:

• Are all four boundaries present — failure onset, first signal, first correct diagnosis, first mitigation?
• Is each timestamp linked to a log, alert history entry, or chat transcript, with a note on any known lag or retention limit?
• Are “page fired,” “acknowledged,” and “first correct diagnosis” separate rows?
• Are the dead hypotheses recorded with the evidence that killed them?
• Does each action item name the gap it closes?
• Is the language factual — roles rather than names, system states rather than judgments?

If the answer to the first three is yes, the timeline will survive review. If the answer to the last three is yes, the action items will survive the next incident.

FAQ

What if we genuinely cannot reconstruct the diagnosis time?

Say so. Label the row “reconstructed from chat, approximate” and give the range. A labeled estimate is more useful than a false precision, and the SRE Workbook’s guidance to provide verifiable data applies to the caveat as much as to the timestamp.

Does this add process overhead during the incident?

No. The boundary rows are filled in after the incident, from logs and chat. The only in-incident change is that someone notes the first correct hypothesis when it is stated, which is usually already happening in the incident channel. If your team uses a scribe role, that is the scribe’s job.

Should we add these fields to our template permanently?

Yes, as empty rows. A template that asks for the diagnosis boundary will get one; a template that does not will get a narrative. The PagerDuty template’s insistence on evidence links is the model to copy.

What about incidents where diagnosis and mitigation overlap?

Record both rows anyway. Overlap is information: it usually means the mitigation was a diagnostic action, such as restarting a process to see whether the symptom clears. That pattern is worth naming in the postmortem because it often indicates a missing observability signal.

A maintenance window is a planned change with a known failure surface. That makes it a good candidate for a pre-mortem: a short exercise in which the team assumes the change has already failed and writes down why. Gary Klein’s description of the technique is blunt about the mechanism — people are reluctant to voice reservations during planning, and a pre-mortem makes it safer for knowledgeable dissenters to speak up (Harvard Business Review, 2007).

For a team of two to fifteen engineers who are their own on-call rotation, the pre-mortem is not a risk register and not a prediction. It is a drill-design tool. Its output should be a set of rehearsable checks with pass/fail results, each tied to one named failure mode. The window gets scheduled when the checks have results, not when the team has promised to be careful.

Frame the exercise before you frame the migration

Klein’s instruction is to assume the project has already failed and then generate plausible reasons. Applied to a PostgreSQL migration, the opening sentence is not “what could go wrong” but “the window is over and it went badly — what happened?” The answers tend to cluster around a small number of surfaces: the backup, the archive, version and extension compatibility, the rollback path, and the people.

Run the session with the people who will actually be on the call. In a small team that is usually everyone who touches the database, plus whoever owns the application that reads from it. One person writes; the writer is not the decision-maker. If the team has no dedicated facilitator, rotate the writing role and keep the decision-maker’s hands off the keyboard so they can listen.

The pre-mortem should be run against the rollback plan as hard as against the forward plan. If the team cannot state in one sentence what “abort” means and who says it, the exercise has found its most important gap before it has found anything else.

Failure mode 1: the backup you would restore from is not the backup you think it is

The first plausible reason the window failed is that the restore did not work. This is the failure mode most worth converting into a drill, because it is cheap to test and expensive to discover at 02:00.

PostgreSQL’s documentation is explicit that recovery from a continuous archive requires a continuous sequence of archived WAL files extending back at least as far as the start time of the backup (PostgreSQL 18, Section 25.3). It also advises setting up and testing the WAL archiving procedure before taking the first base backup. The pre-mortem check that follows is concrete: restore a base backup plus the WAL needed to reach a chosen recovery target into a scratch instance, on hardware that is not the production host, before the window. Record the recovery target, the command used, and the time to completion.

If the team uses pgBackRest, the documentation states that a restore requires the backup files and one or more WAL segments to work correctly (pgBackRest User Guide). That is the minimum the drill must exercise. A restore that only copies files and never starts the cluster has not tested the thing the pre-mortem is worried about.

For file-level backups outside PostgreSQL — configuration directories, application secrets, the runbook repository — the same logic applies. restic’s documentation describes repository integrity checking and snapshot listing, but integrity checking is not the same as a restore drill. The pre-mortem should name which restic snapshots the migration depends on and who will restore one to a scratch path before the window.

Failure mode 2: WAL archiving silently stopped

The second plausible reason the window failed is that the archive had a gap the team did not notice. PostgreSQL’s documentation is precise about the contract: the archive command must return zero exit status if and only if it succeeds; a nonzero status tells PostgreSQL the file was not archived and it will retry periodically (Section 25.3.1). The same section warns that if archiving falls significantly behind, more data would be lost in a disaster, and pg_wal/ can accumulate not-yet-archived segments until disk space is exhausted.

The pre-mortem check is a gate, not a glance. Before the window, confirm that the most recent archived segment is recent, that the archive command’s exit behavior is what the team believes it is, and that the monitoring that would catch a stalled archiver is actually firing. If the team uses pgBackRest, note that the documentation requires local and remote pgBackRest versions to match exactly, and that a mismatch prevents WAL archiving and backups from functioning until the versions match (pgBackRest User Guide). A version skew introduced during migration preparation is exactly the kind of quiet failure the pre-mortem exists to surface.

One more check that belongs here: WAL archiving does not restore changes to postgresql.conf, pg_hba.conf, or pg_ident.conf, because those are edited manually rather than through SQL (Section 25.3). If the migration changes any of those files, the pre-mortem should name where the new versions are stored and how they get onto the restored host.

Failure mode 3: version and extension incompatibility

The third plausible reason the window failed is that the target version would not accept something the source version allowed. PostgreSQL’s documentation notes that for major releases the internal data storage format is subject to change, that dump-and-restore is the traditional method, that pg_upgrade is faster, and that replication methods are also available (Section 18.6). The same section recommends that cautious users test client applications on the new major version before switching over fully, and suggests concurrent installations of old and new versions.

Extension compatibility is where this failure mode usually bites. Azure Database for PostgreSQL flexible server documents upgrade validation checks that evaluate compatibility and configuration without initiating the upgrade, and states that these checks do not modify the server version, trigger downtime, or restart the server (Microsoft Learn). The same documentation states that in-place major version upgrades do not support all PostgreSQL extensions, and that the upgrade fails during precheck if a blocked extension is present on an affected upgrade path. That is a useful model even for teams not on Azure: enumerate every extension, every version-specific blocker, and every object that depends on an extension, and resolve them before the window rather than during it.

The pre-mortem check is an inventory with a result per row: extension name, installed version, target-version support, owner, and disposition. A row without a disposition is a check that has not passed.

Failure mode 4: no tested rollback

The fourth plausible reason the window failed is that the team could not get back. Azure’s documentation is direct about the shape of this problem: after a successful in-place major version upgrade there are no automated ways to revert to the earlier version, though point-in-time recovery to a time before the upgrade can restore the previous version on a new server (Microsoft Learn). Self-managed PostgreSQL has the same property in different clothing: the rollback is a restore, and a restore is a procedure that has to be rehearsed.

The pre-mortem should force the team to write the abort criterion in one sentence and name the person who calls it. It should also force a second sentence: what the rollback actually is — a restore to a new host, a DNS switch back to the old primary, a logical replication reversal — and how long it takes. If the answer is “we would figure it out,” the pre-mortem has found a gap that no amount of scheduling discipline will close.

Failure mode 5: the on-call rotation is the same two people

The fifth plausible reason the window failed is staffing. In a team that is its own on-call rotation, the migration window competes with whatever else pages that night. The pre-mortem should include a staffing failure mode: who is awake, who is the decision-maker, who is the scribe, and what happens if the window overruns into the next working day.

This is a planning argument, not a claim about any specific team. The point is that the pre-mortem is the right place to notice that the person who knows the extension inventory is also the person who would be paged for an unrelated incident at the same hour. Assign roles in advance: operator, scribe, decision-maker. Write a stop-work condition — a time, a metric, or an event — that ends the window regardless of progress.

Convert each failure mode into a check with an owner and a result

The pre-mortem is only useful if it changes what happens before the window. The conversion is mechanical: every plausible failure becomes a check, every check gets an owner and a deadline, and every check returns a pass, a fail, or a not-run. The window is scheduled when every check has a result. A promise to be careful is not a result.

A hypothetical example, clearly labeled as such: a two-person team plans a PostgreSQL major-version window. The pre-mortem lists “the extension we forgot is blocked on the target version” and “the archive command has been failing quietly for a week.” Each becomes a check with an owner and a deadline before the window. The first check is an extension inventory reconciled against the target version’s release notes. The second is a review of the archiver’s recent history and the monitoring that would have caught a stall. The window is scheduled only when both checks return a result. If either fails, the window moves; that is the pre-mortem doing its job.

This is also where the pre-mortem connects to the recovery checklist the team should already have. Writing the recovery checklist before you need it is the same discipline applied to a different artifact: Write the Recovery Checklist Before You Need It describes the checklist as something you build and test in advance, not something you assemble during an incident. The pre-mortem is the moment to confirm that the checklist exists, that it names the current backup repository, and that someone has run it end to end.

After the window: read the postmortem against the pre-mortem

Google’s SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause or causes, and the follow-up actions to prevent recurrence (Site Reliability Engineering, Chapter 15). It also states that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary, and that a blameless postmortem focuses on contributing causes without indicting any individual or team.

The pre-mortem gives the postmortem a comparison set. After the window, the team can ask which predicted failures happened, which did not, and which surprises were absent from the list. Predicted failures that did not occur are as informative as unpredicted ones, because they show where the team’s model of the system is miscalibrated. A pre-mortem that predicted an archive gap and found none is not wasted; it is evidence that the archive check worked, and it should be recorded as such.

The postmortem should also record whether the abort criterion was used, and if not, why. A window that overran because no one was willing to call the abort is a finding about the plan, not about the people.

What the pre-mortem does not do

A pre-mortem reduces surprise. It does not eliminate downtime, and it is not a substitute for a tested restore. It can schedule the restore drill and name its owner, but the drill itself is the evidence. The pre-mortem only ensures the drill happens before the window rather than during it.

It also does not replace the platform’s own validation. If the team runs on a managed service that offers pre-upgrade validation checks, those checks are the authoritative source for that platform’s blockers, and the pre-mortem should consume their output rather than duplicate it. If the team runs self-managed PostgreSQL, the equivalent is the extension inventory, the version-specific release notes, and a restore drill against the actual backup repository.

The useful claim is narrow: a pre-mortem converts a list of worries into a list of checks, and a list of checks with results is a better basis for scheduling a maintenance window than a list of worries. That is enough.

FAQ

How long should a pre-mortem for a database migration take?

Long enough to enumerate the failure modes and assign owners, short enough that the people who will be on the call actually attend. For a small team, the session is usually one meeting plus asynchronous follow-up on the checks. The checks themselves take longer than the meeting; the meeting’s job is to produce them.

Who should facilitate if the team has no dedicated SRE?

Rotate the writing role and keep the decision-maker out of the scribe seat. The facilitator’s job is to keep the exercise in the past tense — “the window failed, why” — rather than letting it drift into a forward-looking risk review. If the team is two people, one writes and one talks, and they swap for the next migration.

Does a pre-mortem replace a restore drill?

No. The pre-mortem can schedule the drill and name its owner, but the drill is the evidence. PostgreSQL’s documentation advises testing the WAL archiving procedure before taking the first base backup; the pre-mortem is the mechanism that ensures the test happens before the window rather than during it.

What if the pre-mortem finds a failure mode the team cannot fix before the window?

Then the window moves, or the scope changes, or the team accepts the risk explicitly and records the acceptance. The pre-mortem’s value is that the acceptance is a decision rather than an oversight. A check that returns “fail, accepted by [name] on [date]” is a result; a check that returns nothing is not.

How does this differ from a standard change-management review?

A change review asks whether the change is approved. A pre-mortem asks what would make it fail and whether the team has rehearsed the answer. The two can share a meeting, but the pre-mortem’s output is a set of checks with results, not an approval.