There is a white rounded banner at the top of the shirt. A small bell icon sits at its left edge. The banner reads, in plain sans-serif: There is motion at your front door. Below it, inside a circular frame tinted green, an orange tabby stares directly into the lens, its face pressed close enough that the porch siding and lights behind it are soft and secondary. The cat is the motion. The notification is the setup. The joke is the gap between them.

We keep coming back to that gap because it is the same gap we spend most of our working hours trying to close. A doorbell camera alert is a terse, automated message. It tells you a sensor tripped. It does not tell you whether the person at the door is a delivery, a neighbor, or the cat. The shirt prints the alert and the answer side by side, and the answer is absurd: the motion was a wide-eyed tabby who has no business ringing anything.

The design is doing something specific. It is not a generic cat meme with a caption bolted on. The composition is a stack: notification chrome on top, camera feed below. That stack is the visual grammar of every monitoring tool we run. A banner, a bell, a timestamp we cannot see but assume, a circular viewport. The green tint reads as night vision or low-light sensor output, though we are interpreting that from the color, not from any printed label. The porch siding and lights behind the cat establish a real exterior. The cat’s proximity to the lens is the punchline: the system detected motion, and the motion filled the frame.

What makes it work as an object rather than just a joke is that the wording is a complete, usable alert. There is motion at your front door is exactly the kind of message a lean team writes when it has thirty seconds to name a condition and no time to explain it. It has a subject, a state, and a location. It does not say what to do. That is the failure mode it accidentally documents: alerts that describe a condition without a next action are technically true and operationally useless. The shirt knows this. The cat is the proof.

We have written before about the difference between a slogan and a usable promise. This shirt sits on the slogan side, deliberately. It is not a runbook. It is not a decision tree. It is a notification that has been stripped of its consequence and handed to a cat. The humor comes from the mismatch between the seriousness of the banner and the silliness of the subject. The banner is doing its job. The subject is not.

If you are the kind of reader who keeps a pager and a doorbell camera, the shirt reads as a small, wearable comment on the alerts you already receive. It is available as a standard t-shirt through the Quirky Cotton Collective listing, where the printed artwork matches the description above: the white banner, the bell, the green-tinted circular view, the orange tabby close to the lens. We are describing what we observed in the artwork, not a wearing experience.

The useful argument here is not that the shirt is funny, though it is. The argument is that the shirt isolates a specific kind of message failure: the alert that tells you what without telling you what next. In our own runbooks, we have a rule that every alert must name a condition, a threshold, and a first action. There is motion at your front door passes the first test and fails the second two. It is a perfect example of why we keep that rule. The cat is not the problem. The cat is the reason the rule exists.

We would not wear this to an incident review. We might wear it to a team lunch where someone has just paged the on-call because a synthetic check went red and the message said only endpoint unreachable. The shirt is a shorthand for that moment. It is a visual joke about the distance between detection and response, printed on cotton.

What we’d do differently: If we were writing the alert for this shirt, we would add a second line: Check the porch. It is probably the cat. That is the difference between a notification and a runbook. The shirt is funnier without it. Our alerts are not.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The question behind the question

“Rotate the backup key” is ambiguous, and the ambiguity is where teams get hurt. In most backup stacks there are at least two layers of encryption: an outer credential that unlocks the repository or wraps metadata, and inner keys that encrypt the actual archive contents. Rotating the outer layer is a credential-lifecycle operation. Rotating the inner layer is a re-encryption project. Confusing the two produces either a false sense of security (you changed a password and believe the archives are re-encrypted) or an unnecessary rewrite of terabytes of immutable history.

For a 2–15 engineer team that is its own on-call rotation, the practical scope is narrower than vendor marketing suggests: issue a new access credential, prove old archives still open, then revoke the old credential. The dangerous failure mode is ordering — removing the old key before a verified restore from an archive created under it. This article maps what each common tool actually protects, then gives a drill-first procedure that keeps old archives readable.

Map the layers before touching anything

Write down, for each backup system you run, which key protects which artifact. The answer differs by tool, and the differences matter.

restic: repository password keys

restic’s key command “allows you to set multiple access keys or passwords per repository” (restic-key(1)). That is the whole rotation model: a repository can have several access credentials, each of which can decrypt the repository’s master key material. Adding a key does not re-encrypt any snapshot data. Removing a key revokes that credential’s ability to open the repository; it does not change the ciphertext of existing snapshots.

One command deserves special caution. restic key passwd “creates a new key, validates the key and removes the old key ID” (restic-key-passwd(1)). It is a combined add-and-remove in a single step. If the new password is not yet stored somewhere your recovery process can reach, you have just removed the working credential. Prefer the explicit add, verify, remove sequence over passwd when the repository is the only copy of production history.

pgBackRest: repository cipher passphrase and subkeys

pgBackRest “will encrypt the repository based on a user-provided password” (pgBackRest User Guide). The configuration option is repo1-cipher-pass, paired with repo1-cipher-type (pgBackRest Configuration Reference). What that passphrase actually protects is narrower than people assume. As the pgBackRest maintainer explained in a 2019 issue thread: “cipher-pass is only used to encrypt backup.info and archive-info. backup.info contains a subkey that encrypts backup.manifest and each backup set has its own subkey” (pgbackrest#680).

That hierarchy is the reason rotation is tractable. The master passphrase wraps small metadata files. The metadata files carry the subkeys that encrypt manifests and WAL. Each backup set already has its own subkey, and per the same thread, the manifest key is rotated on every full backup. The archive key that encrypts WAL is not currently rotated, and a stanza-rekey command is an open feature request, not a shipped feature (pgbackrest#2814). Do not plan around a command that does not exist in your installed version.

The documented-by-maintainer rotation path for the master passphrase is to re-encrypt archive.info and backup.info under a new key. The maintainer described doing this with the openssl command-line tool because the encrypted files are OpenSSL-compatible (pgbackrest#680). That is a manual, high-consequence operation. Treat it as a change with a rollback plan and a restore test, not a config edit.

age: recipients on individual files

age has no repository and no master key. Each file is encrypted to one or more recipients. “Files can be encrypted to multiple recipients by repeating -r/--recipient. Every recipient will be able to decrypt the file” (age README). Identity files “contain one or more secret keys … Multiple key files can be provided, and any unused ones will be ignored” (same source).

This means rotation for age-encrypted archives is a forward-looking operation. New files go to new recipients. Old files remain decryptable only by the identities they were encrypted to. There is no in-place re-wrap. If policy requires old files to be readable only by the new recipient, you must decrypt and re-encrypt them — a real data operation with real cost, not a key-management step.

AWS KMS: key material versions, not data re-encryption

If your backups are wrapped by a KMS key, the rotation semantics are explicit: “Key rotation has no effect on the data that the KMS key protects. It does not rotate the data keys that the KMS key generated or re-encrypt any data protected by the KMS key” (AWS KMS Developer Guide). Decryption is transparent: “When you use the rotated KMS key to decrypt ciphertext, AWS KMS uses the same version of the key material that was used to encrypt it” (same source). AWS KMS retains all key material for a KMS key with AWS_KMS origin even if rotation is disabled, and deletes it only when the key itself is deleted (same source).

The operational consequence: rotating a KMS key does not make old ciphertext unreadable, and deleting the KMS key does. Schedule deletion with the same care you would give to deleting the only copy of a password.

The safe order of operations

Across all four tools, the sequence that preserves access is the same:

  1. Add the new credential or recipient alongside the old one.
  2. Verify that the new credential opens the repository and that the old credential still opens archives created before the change.
  3. Restore a real artifact from an archive created before rotation — not a listing, not a metadata read.
  4. Remove the old credential only after step 3 succeeds and the new credential is stored where your recovery process can reach it.

Steps 1 and 4 are reversible. Step 3 is the one that catches mistakes. A rotation that ends with a successful restic snapshots or pgbackrest info has proven that metadata is readable. It has not proven that the data behind the metadata is recoverable under the new credential arrangement.

restic walkthrough

The exact subcommand syntax for key add and key remove varies by restic version; check restic key --help on the installed binary before scripting. The shape of the operation is stable:

# 1. Add a new access key, reading the new password from a file
restic -r s3:s3.amazonaws.com/example-backups key add \
  --new-password-file /run/secrets/restic-new-password

# 2. List keys and record the new key ID
restic -r s3:s3.amazonaws.com/example-backups key list

# 3. Verify the new credential opens the repository
RESTIC_PASSWORD_FILE=/run/secrets/restic-new-password \
  restic -r s3:s3.amazonaws.com/example-backups snapshots --latest 1

# 4. Restore a file from a snapshot created before rotation
RESTIC_PASSWORD_FILE=/run/secrets/restic-new-password \
  restic -r s3:s3.amazonaws.com/example-backups restore latest \
  --target /tmp/restore-lottery --include /etc/example.conf

# 5. Only after step 4 succeeds, remove the old key by ID
restic -r s3:s3.amazonaws.com/example-backups key remove <old-key-id>

Two notes. First, key passwd collapses steps 1 and 5 into one command and removes the old key ID as part of its operation (restic-key-passwd(1)); avoid it for the primary repository unless you have already stored the new password in your break-glass path. Second, removing a key does not shrink the repository or change snapshot ciphertext. The old snapshots remain encrypted under the repository’s master key material, which the remaining access keys can unwrap. Removing the last access key you control is the only way to lose access.

pgBackRest walkthrough

pgBackRest does not ship a supported rekey command as of the 2.60/2.61 milestone discussion (pgbackrest#2814). The master passphrase rotation is a manual re-encryption of archive.info and backup.info. The maintainer’s description of the mechanism is the authoritative starting point (pgbackrest#680); the exact openssl invocation depends on the cipher type configured in repo1-cipher-type and should be rehearsed against a copy of the repository, not the live one.

Before any manual re-encryption, confirm what you are changing:

  • repo1-cipher-pass wraps backup.info and archive.info.
  • backup.info carries the manifest subkey and per-backup-set subkeys.
  • archive.info carries the archive key used for WAL.
  • Historical WAL and manifests stay under their prior subkeys. Re-wrapping the info files does not re-encrypt them.

That last point is the one to write into the runbook. A reader who assumes rotation re-encrypts WAL will make wrong decisions about retention and about whether a compromised passphrase still exposes old archives. It does, until the subkeys themselves are rotated, and the archive key is not currently rotated by the tool.

PostgreSQL’s own documentation reinforces why the WAL chain must stay intact: “To recover successfully using continuous archiving … you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup” (PostgreSQL 18, Continuous Archiving and PITR). Any rotation procedure that interrupts WAL archiving or changes the archive identity mid-stream risks a gap. The 2019 issue thread documents exactly this hazard when a second repository was introduced with a new key and retention purged WAL prior to the first backup in the new repo (pgbackrest#680). The lesson is not that rotation is unsafe; it is that rotation and repository migration are different operations and must not be combined in one change window.

age and file-level archives

For archives encrypted directly with age, rotation is a recipient change, not a repository change. Add the new recipient to the encryption command for new files, keep the old identity available for decryption, and decide explicitly whether old files need re-encryption.

# New files go to both recipients during the transition
age -r age1newrecipient... -r age1oldrecipient... \
  -o backup-2026-10.tar.gz.age backup-2026-10.tar.gz

# Old files decrypt with the old identity, which is still on disk
age --decrypt -i old-identity.txt backup-2025-03.tar.gz.age > backup-2025-03.tar.gz

“Every recipient will be able to decrypt the file” (age README), so a dual-recipient transition period is safe and reversible. The irreversible step is deleting the old identity before you have confirmed that no archive you still need was encrypted only to it. If policy requires old archives to be readable only by the new recipient, that is a decrypt-and-re-encrypt job with storage and time costs; budget it as such rather than treating it as part of key rotation.

A restore-lottery drill that proves the rotation worked

The verification step in the procedure above should be a real restore, not a metadata query. A restore lottery picks a random archive from before the rotation date and restores it to a scratch location. The drill is small enough to run during an on-call shift and specific enough to catch the failure modes that matter.

  1. Pick a snapshot or backup set created at least one retention cycle before the rotation date.
  2. Restore it using only the new credential. Do not fall back to the old one.
  3. Compare a checksum or a known row count against the expected value recorded at backup time.
  4. Record the archive ID, the restore timestamp, and the credential used in the decision record.

If step 2 fails, the rotation is not complete regardless of what key list or info reports. The recovery checklist article at Write the Recovery Checklist Before You Need It is a useful companion for structuring the drill so it can be executed by whoever is on call, not only by the person who designed it.

Runbook and decision-record notes

Record, at minimum: the key ID or key version, the rotation date, the archive IDs used in the restore lottery, and the name of the person who holds break-glass access to the new credential. The next engineer to answer “which key opens the 2024 archive?” should not have to reconstruct that from chat history. For pgBackRest, note explicitly that the archive key and manifest key were not rotated by the master-passphrase change, so the decision record does not overstate what was accomplished. For restic, note which key IDs remain and which were removed. For age, note which identities are still required to read retained archives.

A rotation that requires a second person who is asleep is a rotation that gets deferred indefinitely. Keep the runbook short enough to execute solo, and keep the break-glass credential in a location the on-call engineer can reach without a meeting.

What not to do

  • Do not remove the old key before a verified restore from an archive created under it. restic key passwd does both in one step; understand that before running it.
  • Do not assume rotation re-encrypts stored data. AWS KMS states it plainly (AWS KMS Developer Guide), and pgBackRest’s subkey hierarchy has the same property.
  • Do not combine key rotation with repository migration or retention changes in one window. The 2019 pgBackRest thread shows how a new repository plus a new key plus retention produced a WAL gap (pgbackrest#680).
  • Do not plan around a stanza-rekey command until it ships in the version you run. It is an open feature request (pgbackrest#2814).
  • Do not delete a KMS key to “rotate” it. Deletion removes the key material; rotation does not (AWS KMS Developer Guide).

FAQ

Does removing a restic key make old snapshots unreadable?

No. Removing an access key revokes that credential’s ability to open the repository. The snapshots remain encrypted under the repository’s key material, which the remaining access keys can unwrap. Access is lost only when no remaining key can open the repository.

Does rotating the pgBackRest repo1-cipher-pass re-encrypt WAL?

No. The passphrase wraps backup.info and archive.info. WAL is encrypted under the archive key carried in archive.info, and that key is not currently rotated by the tool (pgbackrest#2814). Historical WAL stays under its prior key.

Can I decrypt an age file with an identity that is not the current recipient?

Only if that identity was a recipient when the file was encrypted. age has no re-wrap operation; old files keep the recipients they were encrypted to. Keep old identities available for as long as you retain archives encrypted to them.

Does AWS KMS rotation break decryption of old backups?

No. AWS KMS automatically uses the key material version that encrypted the ciphertext when decrypting (AWS KMS Developer Guide). Deleting the KMS key is the operation that removes that ability.

How often should we rotate?

That is a policy decision, not a technical constant. The technical constraint is that rotation must be followed by a verified restore from a pre-rotation archive before the old credential is retired. Choose an interval your team can actually execute end to end, including the restore lottery.

Why a fresh account is the right rehearsal target

A restore drill that runs inside the production account is a weaker test than one that runs in a fresh account. The production account silently supplies IAM roles, VPC defaults, DNS, and credentials that a real disaster may not. When you rehearse into a new account, you are testing the parts of recovery that are usually invisible until they fail: identity, network reachability, key material, and the operational knowledge of the person holding the pager at 03:00.

This is a drill-first guide for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated SRE department. The target is a PostgreSQL cluster restored from a pgBackRest repository into a fresh AWS account in a different region. The same reasoning applies to restic repositories and to bare-metal targets, with the tool-specific mechanics called out where they differ.

The argument is simple: treat the fresh account as a hostile environment. Assume nothing from production is present. Rebuild the backup path from scratch, verify the restore against the backup tool’s own documented mechanics, and record what the drill did not prove.

Define success before touching anything

A restore command that exits zero is not a successful drill. The pass criterion should be a verification query against the restored database: the cluster is out of recovery, the recovery target timestamp is known, and row counts on a few named tables match expectations within a tolerance you set in advance.

Write the criterion down before the drill starts. A workable example:

  • A PostgreSQL instance is running in the fresh account and region.
  • pg_is_in_recovery() returns false.
  • The recovery target timestamp actually reached is recorded.
  • Row counts for three named tables are within a stated tolerance of the source at the chosen target time.
  • The restore host’s pgBackRest version matches the version that wrote the archive.

Everything else — DNS cutover, application reconfiguration, credential rotation — is out of scope for this drill and belongs in the decision record as unproven.

Step 1 — Provision the target account deliberately

Create the recovery account as a member account in your AWS organization rather than as a standalone account. AWS Organizations automatically creates the IAM role OrganizationAccountAccessRole in the member account, which enables users and roles in the management account to exercise full administrative control over it. That role is the documented bootstrap path; decide in advance whether you will use it or replace it with a narrower break-glass role.

Two consequences matter for a drill. First, policies attached to the organization root immediately apply to all users and roles in the newly created account. If your root has service control policies that restrict regions or services, the recovery account inherits them. That is usually desirable, but it is a constraint you should know about before the drill, not during it. Second, the account is created in the root of the organization and can be moved between organizational units afterward; place it where its SCPs match the recovery intent.

Record the account ID, the region, and the role name in the drill’s decision record. The account is an artifact of the drill, not a permanent fixture, and it should be tearable down without ambiguity.

Step 2 — Rebuild the backup path from scratch

The restore host in the fresh account must reach the repository. For pgBackRest that means network reachability to the repository host or object store, the repository password or encryption key, and a pgBackRest binary of the correct version.

Version matching is not optional. The pgBackRest user guide states that the local and remote pgBackRest versions must match exactly and should be upgraded together; if there is a mismatch, WAL archiving and backups will not function until the versions match. A restore host running a newer pgBackRest than the repository host is a failure mode worth rehearsing deliberately rather than discovering during an incident.

Stage the repository password or encryption key outside the production account. If the only copy lives in a production secrets manager that the fresh account cannot read, the drill has already found a real gap. The same reasoning applies to restic: the repository password is required to open the repository, and a copy that exists only in production is not a recovery asset.

For restic repositories, note that the copy command reads and uploads the entire snapshot because the source and destination repositories use different encryption keys, which may incur higher bandwidth usage and costs than normal backup runs. It also does not re-chunk files, which may break deduplication between copied files and files already in the destination repository. If your drill includes moving a restic repository into the fresh account, budget for the full transfer and do not assume deduplication will hold across the copy.

Step 3 — Choose the restore mode that matches the drill

pgBackRest supports several restore modes, and the drill should name which one it is testing.

  • Full restore to latest. For a standard non-delta restore, the data directory must be empty. This is the simplest mode and the right first drill.
  • Specific backup set. By default pgBackRest restores the most recent backup; --set restores a specific backup label. Use this when you want to prove that an older backup is still restorable.
  • Delta restore. Compares checksums against the backup manifest and replaces only files that differ. Faster for large clusters, but it assumes a data directory already exists, so it is a second drill, not a first.
  • Point-in-time recovery. Replays WAL to a target timestamp. In pgBackRest, --target-action=promote causes PostgreSQL to promote to a primary after reaching the recovery target; without it, the cluster pauses in recovery at the target point and waits for manual intervention. A drill that omits the promote action will appear to hang, which is a useful thing to see once and then never again.

PostgreSQL’s own continuous-archiving documentation is worth reading alongside the tool docs. It notes that pg_dump and pg_dumpall do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. If your recovery plan depends on logical dumps, the drill will not exercise WAL replay, and the pass criterion should say so.

The same documentation notes that WAL archiving will not restore changes made to configuration files such as postgresql.conf, pg_hba.conf, and pg_ident.conf, since those are edited manually rather than through SQL operations. In a fresh account, those files must come from somewhere else — a configuration repository, a secrets store, or a documented reconstruction step. The drill should prove that the somewhere else exists.

Step 4 — Handle what breaks in a fresh account

Three categories of failure are common when restoring into a different host or directory layout, and all three are documented pgBackRest behaviors rather than surprises.

Tablespace paths

When restoring to a different host or directory layout, tablespace paths from the original cluster may not exist. Use --tablespace-map to redirect individual tablespaces, or --tablespace-map-all to redirect all tablespaces to a single directory. Rehearse the flags before the drill, not during it.

Symlinked directories

pgBackRest records symbolic links inside the data directory in the backup manifest and recreates them on restore. If pg_wal points to a separate disk on production, the restore host needs a corresponding path or a --link-map to redirect it. A restore that fails because /mnt/wal-disk does not exist in the fresh account is a configuration gap, not a backup failure.

File ownership

For a non-root restore, all restored files belong to the user and group running pgBackRest. If existing files have different ownership and the executing user cannot change it, the restore fails. For a root restore, pgBackRest recreates the ownership recorded in the backup manifest, but only user and group names are stored, not numeric IDs, so the same names must exist on the restore host. Decide which model the drill uses and make the user and group names match.

Step 5 — Verify against the database, not the exit code

Connect to the restored cluster and run the checks you wrote down in step zero. A minimal sequence:

SELECT now(), pg_is_in_recovery();
SELECT schemaname, tablename, n_live_tup
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC
LIMIT 20;

Record the recovery target timestamp actually reached. If the drill used PITR, compare it to the intended target. If the drill restored to latest, record the timestamp of the most recent WAL segment replayed. Either way, the number goes in the decision record.

If the drill uses the Crunchy Postgres Operator, the documentation describes bootstrapping a new cluster from a pgBackRest repository via spec.dataSource as the recommended way to stand up a clone or replacement cluster from an off-site backup. The same documentation warns that if the original primary cluster is still running when a standby is promoted, you will have two writable clusters pointing at the same data, and advises ensuring the original primary is offline or network-isolated before promotion. In a fresh account, the original primary is not reachable by default, which is one of the reasons the fresh account is a good rehearsal target — but the split-brain warning still belongs in the runbook for the real event.

Step 6 — Tear down and write the decision record

The drill is not finished until the decision record exists. Put it where the runbooks live, not in a chat thread. A useful record answers four questions:

  • What did the drill prove? Name the restore mode, the backup set or target timestamp, and the verification queries that passed.
  • What did it not prove? DNS cutover, application reconfiguration, credential rotation, and any service outside PostgreSQL are the usual omissions. Say so explicitly.
  • Which runbook step was wrong? If the drill required a flag that the runbook does not mention, the runbook is the defect.
  • What is the next drill’s target? A delta restore, an older backup set, a PITR to a named restore point, or a restic repository opened from the fresh account.

This is the same discipline as a blameless postmortem, applied before the incident. The recovery checklist is the natural companion artifact: the drill tests the checklist, and the checklist records what the drill found.

A short pre-mortem for the next drill

Before scheduling the next drill, ask what would make it fail silently. A few prompts that have earned their place:

  • If the repository password were rotated between drills, would the fresh account still open the repository?
  • If the pgBackRest version on the restore host drifted, would the drill detect the mismatch before the restore command, or after?
  • If the target timestamp were earlier than the selected backup’s stop time, would the operator recognize the failure as a target-selection problem rather than a corrupt backup?
  • If the person running the drill were not the person who wrote the runbook, would the runbook still be sufficient?
  • If the fresh account’s SCPs blocked a service the restore needs, would the error message name the SCP or the service?

None of these require new tooling. They require that the drill be treated as a test of the team’s knowledge, not only of the backup.

FAQ

Why not rehearse in the production account with a separate VPC?

Because the production account supplies IAM roles, DNS, and credentials that a real disaster may not. A separate VPC inside the same account tests network configuration but not account-level identity and key access. The fresh account is the harder test, and the harder test is the one worth scheduling.

Do we need a full copy of the repository in the fresh account?

Not necessarily. The restore host can read from the existing repository if network reachability and credentials allow it. What must exist in the fresh account is the ability to authenticate to the repository and the pgBackRest binary of the matching version. For restic, the repository password is the equivalent requirement.

How often should the drill run?

Often enough that the runbook is current and the on-call rotation has run it at least once. For a small team, once per quarter per critical service is a reasonable starting cadence; the decision record from each drill should state whether the cadence still fits.

What about RDS or S3 backups?

This drill is written around pgBackRest and PostgreSQL WAL because those mechanics are documented in the sources cited here. If your recovery path uses managed snapshot services, the drill should be rewritten around those services’ documented restore procedures, and the pass criterion should still be a verification query rather than a console success message.

Sources

A third-party API starts returning 429s. Your queue backs up, your retries pile on, and by the time someone notices, the on-call engineer is staring at a dashboard that says “rate limited” and a vendor status page that says “all systems operational.” The incident ends. Now you have to write it up.

The temptation is to file a short postmortem that says “vendor throttled us, we backed off, resolved.” That document will not help the next on-call engineer, and it will not prevent the next incident. The useful postmortem for a rate-limit incident records three things the team can actually verify: what the dependency did, what your own client did, and what you changed afterward. Everything else is narrative.

This is written for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated reliability staff. The structure below assumes you already write postmortems; if you are still deciding what belongs in one, Google’s Postmortem Culture chapter defines the baseline: a postmortem is “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” A rate-limit incident fits that definition, but each section needs different evidence than a code-defect incident does.

Start with impact and timeline, not with the vendor

Record what users experienced before you record why. Which internal services degraded, for how long, and what the user-visible symptom was. A batch pipeline that fell two hours behind is a different incident from a checkout flow that returned errors for eleven minutes, even if both trace to the same 429.

The timeline needs two specific timestamps that teams often skip: when the first rate-limit response appeared, and when a human first knew about it. The gap between those two is your detection latency, and it is usually the most actionable number in the document. If the first 429 arrived at 02:14 and the first page fired at 02:51, the postmortem should say so plainly, because that thirty-seven-minute gap is a monitoring problem you can fix independently of the vendor.

Also record how the incident was detected. An alert on error rate, a synthetic check, a customer report, and a colleague noticing a stale dashboard are four different detection paths with four different remediation stories. Name the one that happened.

Log the dependency’s observed behavior verbatim

This is the section that saves the next on-call engineer the most time, and it is the one most often written from memory. Do not write it from memory. Paste the actual response.

For each rate-limit response you captured, record:

  • The HTTP status code and the full response body or error payload, verbatim.
  • Any rate-limit or retry headers the vendor returned, with their values.
  • The request ID or trace ID from the response, if the vendor provides one, so a support ticket can reference it.
  • The timestamp of the response, in UTC, and the endpoint or method that triggered it.

The payload matters because “429” is not a complete description. Google’s API design guidance, AIP-193, shows a 429 response carrying a RESOURCE_EXHAUSTED status and a machine-readable ErrorInfo block with a reason and domain. A vendor that follows that pattern gives your client something to branch on. A vendor that returns a bare 429 with an HTML error page gives you nothing, and that difference belongs in the postmortem because it shapes what your retry logic can safely do.

Then record what the vendor’s public documentation says about throttling, and label it as documentation rather than observation. AWS’s API Gateway documentation, for example, states that throttles and quotas “are applied on a best-effort basis and should be thought of as targets rather than guaranteed request ceilings,” and that clients may receive 429 responses they can resubmit “in a way that is rate limiting.” That is a documented behavior, not something you observed in your incident. Keep the two categories separate in the write-up. If the vendor documents a token-bucket model, say so and cite the page. If you are inferring the model from response patterns, say that instead.

Record your own request pattern with numbers

The vendor’s behavior is half the story. Your client’s behavior is the half you can change, and it needs numbers.

Capture the request rate your service was sending at the time of the incident, and the rate in the hour before. Note whether the traffic was steady or bursty, and what caused any change: a scheduled batch job, a retry storm, a traffic spike, a deploy that changed concurrency. If you have per-endpoint or per-tenant breakdowns, include them, because a single noisy tenant can consume a shared quota and the fix may be per-tenant limiting rather than a global change.

Then document the retry logic as it actually existed, not as you intended it to work. Specifically:

  • Were retries bounded, and at what count or duration?
  • Did retries use exponential backoff, and did they include jitter?
  • Did the client honor any Retry-After header the vendor sent?
  • Did retries run concurrently across workers, so that each worker’s backoff was independent and the aggregate rate stayed high?
  • Did the retry logic distinguish a 429 from a 500, or did it treat all failures the same way?

Retries deserve their own subsection because they can turn a rate-limit incident into a self-inflicted one. If your client retried immediately on 429, the postmortem should say so and should show the request rate before and after retries began. That is an observation about your system, not a judgment about the engineer who wrote the retry loop.

Separate the trigger from the contributing conditions

A rate-limit incident rarely has a single root cause, and forcing one into the document produces a misleading record. Write down the trigger and the contributing conditions as separate lists.

The trigger is the event that started the incident: a traffic spike, a batch job that doubled the request rate, a vendor-side quota change, a retry storm. The contributing conditions are the properties of your system that let the trigger become an incident: no client-side limiter, no alert on 429 rate, no documented vendor quota in the runbook, no circuit breaker between your service and the dependency.

This split is what keeps the postmortem useful. “Vendor rate-limited us” is a trigger with no contributing conditions attached, and it produces no action items. “A batch job doubled our request rate; we had no client-side limiter and no alert on 429 rate” produces two fixes the team controls.

It also keeps the document blameless in the specific way that matters here. Google’s guidance is that a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.” A vendor is a team. The postmortem should describe what the vendor’s API did, with evidence, and should not characterize the vendor’s motives or competence. The same rule applies to the engineer who wrote the retry loop two years ago.

Label inferences about the vendor explicitly

You can observe the vendor’s responses. You cannot observe the vendor’s internal throttling state, its quota accounting, or why a particular request was rejected. When the postmortem needs to explain vendor behavior beyond what the response and documentation show, mark it as an inference.

A workable convention: put observed facts in the main text with a timestamp or a pasted payload, and put inferences in a clearly headed subsection such as “What we believe happened on the vendor side.” That subsection can say “we infer the quota is shared across our three API keys because all three began returning 429s within the same minute” without asserting it as fact. If a support ticket later confirms or contradicts the inference, update the postmortem and note the update date.

Record vendor communication and support tickets

If the team opened a support ticket, record the ticket number, the time it was opened, the time of the first response, and the resolution or current status. If the vendor’s status page was updated during the incident, note the timestamp of the update and what it said. If the team requested a quota increase, record the request date and the outcome.

This section is short but it prevents a common failure: six weeks later, nobody remembers whether the quota increase was approved, and the next incident repeats the discovery. If your contract includes service credits or SLA terms that the incident might qualify for, note the relevant clause and whether the team filed a claim. Do not estimate the credit amount in the postmortem; record the clause and the filing status, and let finance or the account owner handle the number.

Write action items with one owner and a tracking reference each

The action items are where the postmortem either changes something or becomes a document nobody reads again. Google’s SRE Workbook is direct about the failure mode: “Action items without clear owners are less likely to be resolved. It’s better to have a single owner and multiple collaborators.” The same chapter criticizes action items that use vague verbs like “improve” or “make better,” and notes that items tagged with equal priority give no guidance on what to do first.

For a rate-limit incident, split the action items into two groups.

Preventative items change the system so the incident is less likely or less severe:

  • Add a client-side rate limiter with a configured ceiling below the vendor’s documented quota.
  • Add exponential backoff with jitter to the retry path, and honor Retry-After when present.
  • Add an alert on the 429 rate, or on the ratio of 429s to total requests, with a threshold tied to your observed baseline.
  • Add a circuit breaker that stops sending requests when the 429 rate crosses a threshold, so the dependency can recover.
  • Document the vendor’s quota and burst behavior in the runbook, with a link to the vendor’s documentation page.

Mitigative items shorten the next incident:

  • Add a runbook entry describing how to identify a rate-limit incident, how to reduce request rate quickly, and how to open a vendor ticket.
  • Add a dashboard panel showing request rate, 429 rate, and retry count side by side.
  • Record the vendor’s support escalation path and the account identifier needed to open a ticket.

Each item gets one owner, a priority, and a tracking reference such as an issue number. If your team uses a postmortem template, the template should have columns for all three, because a table with an empty owner column is easier to notice than a paragraph that omits one.

What to leave out

Three things do not belong in this postmortem.

First, animated language. The SRE Workbook’s critique of a bad postmortem notes that “animated language and dramatic descriptions of events distract from the key message and erode psychological safety.” “The vendor completely failed us” and “the retry loop was obviously broken” are both animated, and both make the document harder to act on.

Second, claims about the vendor’s internal behavior that you cannot support. If the vendor’s documentation does not state a quota, do not write a number as if it were documented. Write what you observed and label the rest as inference.

Third, a single root cause when the evidence supports several contributing conditions. The trigger-plus-conditions structure is more accurate and more actionable than a one-line cause statement.

A constructed example

The following is an illustration, not a case record. A service calls a vendor API at roughly 50 requests per second with no client-side limiter. The vendor’s documentation states a steady-state limit of 40 requests per second and a burst of 100. A nightly batch job starts and doubles the request rate to about 100 per second. The vendor returns 429s with a JSON body containing a Retry-After header of 2 seconds. The client’s retry loop ignores the header and retries immediately, pushing the aggregate rate higher. The first 429 appears at 02:14; the on-call engineer is paged at 02:51 by an alert on queue depth, not on 429 rate.

The postmortem for that incident would record the pasted 429 payload and the Retry-After value, the request rate before and after the batch job started, the retry behavior including the ignored header, the detection gap of thirty-seven minutes, and action items for a client-side limiter, jittered backoff that honors Retry-After, and an alert on 429 rate. It would note that the vendor’s documented burst of 100 was exceeded and that the team inferred the batch job was the trigger, with the batch job’s start time as evidence. It would not say the vendor “throttled us unfairly” or that the engineer who wrote the retry loop “should have known.”

A checklist to paste into your template

The next time a third-party rate limit causes an incident, the postmortem should answer these questions:

  • What did users experience, for how long, and which internal services were affected?
  • When did the first rate-limit response appear, and when did a human first know?
  • What was the exact status code, response body, and any retry headers, pasted verbatim?
  • What does the vendor’s public documentation say about throttling, and what did we infer beyond it?
  • What was our request rate before and during the incident, and what changed it?
  • What did our retry logic actually do, including whether it honored Retry-After and whether it used jitter?
  • What was the trigger, and what were the contributing conditions?
  • What vendor communication happened, with ticket numbers and timestamps?
  • Which action items are preventative, which are mitigative, and who owns each one?

If your team is still building the habit of writing recovery procedures down before an incident, the companion piece on writing the recovery checklist before you need it covers the runbook side of the same problem. The postmortem records what happened; the runbook is what the next on-call engineer reads at 02:14.

A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent recurrence, per Google’s SRE Book. The same chapter states the primary goals plainly: document the incident, understand the contributing causes, and — especially — put effective preventive actions in place. The document is not the deliverable. The closed, verified action item is.

For a team of two to fifteen engineers who are their own on-call rotation, the failure is rarely in the writing. It is in the handoff. During the incident, there was an incident commander, a timeline, and a resolution. The action item that replaces it enters a backlog with no owner, no date, and no definition of done, while feature work enters with all three. Losing that prioritization contest is the default, not a discipline failure.

The gap is structural, not cultural

Google’s SRE Workbook, in its postmortem culture chapter, walks through a deliberately bad postmortem and names the failure modes. The action items are mostly mitigative; the one “preventative” item suggests making humans less error-prone, which the chapter says is less reliable than changing automated systems and processes. Two items use vague language — “Improve” and “Make better” — which the chapter says makes success criteria hard to measure. Only one item was assigned a tracking bug, and the chapter notes that without a formal tracking process, action items from postmortems are often forgotten, resulting in outages.

The same chapter quotes Ben Treynor Sloss: “To our users, a postmortem without subsequent action is indistinguishable from no postmortem.” It also states that action items without clear owners are less likely to be resolved, and that a single owner with multiple collaborators beats shared ownership. That is the whole argument in two sentences. The rest is mechanics.

Atlassian’s incident postmortem guide makes the same point from the planning side: findings should feed back into the planning process so remediation work is prioritized alongside other work. It also says an unreviewed postmortem “might as well never have been written.” Review is necessary but not sufficient. An item can be reviewed, approved, and still rot.

Two failure modes that let items rot

Owner: “the platform team.” A team is not an owner. A team is a distribution list. When the item is due and nothing has happened, there is no name to ask. The Google Workbook’s guidance is explicit: one owner, multiple collaborators. The named owner does not have to do most of the work. The name exists so that escalation has an address.

Done: “merged.” A merged pull request proves code changed. It does not prove the change would have caught the incident. The distinction matters most for detection items — a new alert that fires in staging but pages nobody in production is merged and useless. Verification has to be chosen when the item is filed, while the person who wrote it still remembers what would have caught the failure.

The five-field action item

Every item needs five fields. If one is hard to fill in, the item is not ready to leave the postmortem meeting. This is a recommendation, not a published standard; it borrows the actionable/specific/bounded framing that PagerDuty’s postmortem writing guide attributes to Google’s Lunney, Lueder, and Beyer, and adds a verification field because “bounded” alone does not say how you will know it worked.

  1. Classification. Guardrail (prevents the failure), detection (catches it sooner), process (changes how people respond), or knowledge (spreads what one person learned). The class determines what verification looks like.
  2. One named owner. A person, not a team. Collaborators are listed separately.
  3. Smallest shippable version. The version that could merge this sprint. A rate limit on one endpoint, not “redesign the throttling layer.” Large items get split, not deferred.
  4. Verification method. How you will know it would have caught the incident: replay the trigger, inject the fault, or point at the alert that now fires. “Merged” is not verification.
  5. Decision date. A calendar date written on the item the day it is filed. Default to the end of the next sprint for guardrail and detection changes. Anything further out needs the reason written beside the date.

PagerDuty’s postmortem template shows a lighter version of the same discipline: each action item is a JIRA ticket tagged with the incident date and severity so it can be found and reported later. The tag is the trigger. Without it, the item is a sentence in a document that nobody queries.

A review cadence that is not the incident process

The incident process ends when the postmortem is published. If follow-through depends on the next incident to prompt a review, the cadence is set by outages. That is the wrong clock.

For a small team, the cheapest structural fix is not new tooling. It is a recurring fifteen-minute review attached to a meeting that already exists — the weekly ops sync, the sprint planning opener, the Monday standup that already runs long. The agenda is one filtered list: open action items past their decision date, sorted oldest first, with the owner’s name on each row. A saved filter on the due-date field does this in Jira, GitHub Issues, or Linear without a new system.

Two rules keep the meeting short. Aging items go first, not newest. And an item that has slipped twice gets either a new date with a written reason or an explicit decision to drop it and accept the risk. Silent slippage is how a backlog becomes a graveyard.

Closure on evidence, not on a status field

An item closes when the verification method has been run and the result is recorded on the item: the replay output, the alert that fired, a link to the new test. Marking it done is not closing it. This is stricter than most small teams operate, and it is the step that prevents the “said it was done, nothing actually happened” failure.

For knowledge items — a runbook update, a decision record — verification is a review by someone who was not on the incident. If the runbook cannot be followed by a teammate who was not there, it is not done. This connects directly to the practice of writing the recovery checklist before you need it; the Recovery Checklist article covers the document side of the same problem.

A quarterly recurrence review

Once a quarter, list the incident classes from the previous four quarters and mark each one recurred or did not. A recurrence means the fix was wrong or never shipped. No recurrence tells you which category of fix is worth repeating. This is the feedback loop on the action-item process itself, and it is the only way to learn whether your guardrails or your detection changes are actually buying reliability.

The Google Workbook’s rack-decommission case study is a useful reference point here. Three years after the original outage, a similar incident occurred, and the chapter states that the action items implemented from the original postmortem dramatically reduced the blast radius and rate of the second incident. That is what a shipped, verified fix looks like from the outside: the same class of failure returns smaller.

A worked hypothetical for a small team

The following is a constructed example, not a real incident record. It illustrates how the five fields change what happens after the third postmortem.

A six-person platform team runs a webhook delivery queue. It overflows three times in a year. Each incident gets a competent postmortem. The first two produce action items like “improve queue monitoring” and “add backpressure,” owned by “the platform team,” with no date. Neither ships. The third postmortem opens with an engineer copying from the first.

Under the five-field format, the third postmortem produces three items instead:

  • Detection: alert when queue depth exceeds 80% of the worker pool’s drain rate for five minutes. Owner: one named engineer. Verification: replay the prior incident’s traffic pattern in staging and confirm the alert fires. Decision date: end of next sprint.
  • Guardrail: per-tenant rate limit on webhook enqueue. Owner: a second named engineer. Verification: load test at 3x peak with one tenant saturated; confirm other tenants’ delivery latency stays within SLO. Decision date: two sprints out, with the reason written beside it.
  • Knowledge: runbook section for queue saturation, including the drain command and the escalation path. Owner: the on-call engineer who worked the incident. Verification: a teammate who was not on the incident follows the runbook in staging without help. Decision date: one week.

The difference is not effort. It is that each item now has an address, a date, and a test that can fail. The weekly review surfaces the rate-limit item when it slips. The quarterly recurrence review asks whether the queue class returned after the alert shipped. If it did, the detection fix was the wrong category, and the next postmortem should say so.

What this costs a lean team

Fifteen minutes a week, one saved filter, and the discipline to refuse an action item that has no owner or no verification. The tradeoff is real: some items will be dropped rather than carried indefinitely, and dropping them is an explicit risk decision that should be recorded. That is cheaper than a backlog of items nobody believes will ship, and it is the only version of follow-through that survives a team with no dedicated reliability staff.

FAQ

What should a postmortem action item include?

A classification, one named owner, the smallest shippable version of the fix, a verification method, and a decision date. An item missing an owner or a date is a suggestion, not an action item.

Why do postmortem action items never get done?

Because the incident process ends when the document is published, and implementation then depends on backlog prioritization, where action items rarely win against feature work. The fix is structural: a recurring review outside the incident process, aging surfaced first, and closure on verification evidence.

Doesn’t a named owner conflict with blameless postmortems?

No. Blamelessness is about how the incident is analyzed — focusing on contributing causes rather than indicting individuals. Ownership is about who carries the follow-up work. The Google Workbook treats these as separate concerns and recommends a single owner with collaborators for each action item.

Do we need a new tool?

Usually not. A saved filter on the due-date field in whatever tracker you already use is enough to run the weekly review. The failure mode is not tooling; it is the absence of a cadence and the absence of a verification field.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

The shirt reads I WEAR THIS SHIRT PERIODICALLY, and beneath that line sit four periodic-table-style tiles: 53 I, 74 W, 22 Ti, 16 S. The tile symbols are I, W, Ti, S. They are not a literal spelling of the sentence above them. That gap is the joke, and it is also the most interesting thing about the design.

Most slogan shirts ask you to read a sentence and agree or laugh. This one asks you to read a sentence, then read a second, smaller system underneath it, then notice that the two do not match. The tiles are not an abbreviation of the words. They are a chemical spelling of a different phrase — I W Ti S — that happens to sound like I wear this when you say the element symbols aloud. The word periodically then lands as a separate double meaning: it refers to the periodic table that supplies the tiles, and to the frequency with which the wearer puts the shirt on. The design works because the viewer has to do the conversion. The shirt is not a billboard. It is a small puzzle with a punchline.

What the tiles actually say

Read literally, the four tiles are:

  • 53 I — iodine
  • 74 W — tungsten
  • 22 Ti — titanium
  • 16 S — sulfur

The symbols spell I W Ti S. Say that out loud and you get something close to I wear this — a compressed, element-flavored version of the sentence printed above. The word periodically then lands as a double meaning: it refers to the periodic table that supplies the tiles, and to the frequency with which the wearer puts the shirt on. The design is doing two things at once, and it is doing them with the same four tiles.

That is a more disciplined piece of graphic design than most pun shirts manage. The tiles are not decoration. They are the mechanism. Remove them and the sentence is just a mild boast. Keep them and the sentence becomes a small act of translation.

Why the mismatch matters

The common failure mode in novelty design is over-explanation. A shirt prints the joke, then prints the explanation of the joke, then prints a small arrow pointing at the explanation. The viewer is not asked to do anything. The shirt is a label.

This design avoids that. It prints the sentence and the tiles and stops. The viewer who knows the periodic table gets the conversion immediately. The viewer who does not gets a slightly mysterious set of numbered boxes and a sentence that almost explains them. Either way, the shirt does not condescend. It assumes the wearer is the kind of person who will enjoy the gap between the two readings.

That assumption is the same one we make about our readers. A runbook that explains every command is a tutorial. A runbook that names the failure mode and points at the recovery step assumes a competent operator who will read the rest. The shirt is closer to the second kind of document. It gives you the pieces and trusts you to assemble them.

The difference between a slogan and a usable promise

There is a useful distinction here for anyone who writes internal documentation, on-call handoffs, or team charters. A slogan is a sentence that sounds true. A usable promise is a sentence that tells you what will happen next. The shirt’s top line — I wear this shirt periodically — is a slogan. The tiles underneath are the promise: here is the actual mechanism, here is what the symbols are, here is the conversion you need to perform.

Most team charters fail the same way. They print the slogan — we value reliability, we own our incidents — and never print the tiles. The reader is left with a sentence that sounds true and no way to check it. The shirt, whatever else it is, does not make that mistake. It shows its work.

We keep a small decision log for exactly this reason. When we write down a policy, we try to write down the mechanism next to it: who triggers it, what command runs, what the rollback is. The slogan goes in the title. The tiles go in the body. If we cannot fill in the tiles, the slogan is not ready to ship.

What the shirt is not

It is worth being clear about what this design does not claim. It does not claim the wearer is a chemist. It does not claim the shirt improves anything, cures anything, or signals membership in a profession. It is a pun about the periodic table and the word periodically, printed on a shirt. The tiles are a visual mechanism, not a credential.

That restraint is part of why it works. The shirt is not trying to be a personality test. It is trying to be a small, well-made joke that rewards a second look. The viewer who reads the tiles and says oh, I W Ti S has done exactly the work the design asked for. The viewer who does not still gets a sentence that makes sense on its own.

Where the design sits in the wardrobe

We have written before about shirts that function as boundary markers — the Not Friendly Do Not Touch shirt, the meh shirt, the composition notebook shirt. Those designs work because they say something about the wearer’s relationship to attention, effort, or interruption. This one is different. It is not a boundary. It is an invitation to a small piece of wordplay, worn in the same spirit as a band shirt or a conference shirt: a signal that the wearer enjoys a particular kind of puzzle.

That makes it a reasonable off-duty object for technical readers, not because it demonstrates engineering competence, but because it rewards the same habit — reading a system closely, noticing that the labels do not match the sentence, and enjoying the conversion. The shirt is available on Amazon at the Quirky Cotton Collective listing if you want to read the tiles yourself.

What we’d do differently

If we were designing a shirt like this, we would resist the urge to add a fifth tile or a caption explaining the pun. The design works because it stops at four tiles and one sentence. The temptation in our own documentation is the same: to add one more clarifying paragraph, one more diagram, one more note that explains the note. The shirt is a reminder that the gap between the sentence and the tiles is the point. Fill it in and the joke disappears. We would leave the gap.

The first thing we notice is the triangle. A yellow caution triangle, the kind that usually means wet floor or high voltage, sits above three lines of block lettering: NOT FRIENDLY, DO NOT TOUCH. Below the words, a hand is crossed out — a simple pictogram of a palm with a diagonal slash through it. The whole composition is borrowed from industrial signage, the visual grammar of hazard warnings, and it is doing something slightly different here. It is not warning you about a machine. It is warning you about a person.

That shift — from equipment to human — is the entire joke, and also the entire argument. The shirt is not a slogan in the usual sense. A slogan makes a claim about the wearer’s mood or taste. This is closer to a boundary marker: a terse, legible statement of access policy, printed in the one visual language most people already know how to read without thinking. We keep coming back to that distinction because it matters for how the object works in practice. A shirt that says not friendly in plain text is a joke. A shirt that says it inside a caution triangle, with a crossed-out hand, is a joke that has been formatted as an instruction.

What the sign language actually does

Industrial signage is designed to be parsed at a glance, by people who are not paying attention, under conditions where a mistake has consequences. The caution triangle, the crossed-out hand, the all-caps imperative — these are not decorative choices. They are a compressed protocol. When you put that protocol on a t-shirt, you are borrowing its authority. The wearer is not just saying leave me alone; they are saying it in the register of a system that expects to be obeyed.

We think that is why the design reads as more than a novelty. The crossed-out hand is the key element. It is not a hand waving, not a handshake, not a hand held up in a stop gesture. It is a hand with a slash through it — the universal symbol for this action is prohibited. The shirt is not describing a personality trait. It is prohibiting a behavior. That is a meaningful difference, and it is the difference between a shirt you wear to be funny and a shirt you wear to be left alone.

We should be careful here. We are reading the artwork, not making a claim about the wearer’s actual social preferences. The design is a visual argument, not a diagnostic tool. But the argument it makes is coherent: it takes the grammar of a hazard sign and applies it to the hazard of unwanted interaction. Whether that lands as funny, rude, or simply accurate depends entirely on the room.

The difference between a slogan and a usable promise

In our day jobs we spend a lot of time thinking about the difference between a statement and a procedure. A runbook that says restore from backup is a slogan. A runbook that says run this command, check this output, escalate to this person if it fails is a usable promise. The gap between those two things is where most operational failures live.

The Not Friendly Do Not Touch design sits on the usable-promise side of that gap, at least visually. It does not ask the reader to infer the wearer’s boundaries from context. It states them in the most legible format available. The caution triangle is a trigger. The crossed-out hand is the action. The lettering is the policy. It is, in its own small way, a well-designed interface: the reader knows exactly what is expected of them within about half a second of looking at it.

That is not nothing. Most social signals are ambiguous by design. This one is not. Whether you find that refreshing or abrasive probably depends on how much of your day you spend explaining boundaries that you wish were already obvious.

Why the warning-sign format is the right container

We have written before about shirts that use diagram language — the ramp-and-ball physics pun, the composition notebook surface — and the common thread is that the visual format does most of the work. A joke printed in plain text has to be read. A joke printed as a diagram has to be recognized. Recognition is faster and, for a certain kind of wearer, more satisfying.

The warning-sign format is particularly well suited to this message because it carries an implicit this is not personal clause. A hazard sign is not angry at you. It is not judging you. It is simply informing you that a condition exists and that a behavior is prohibited. When the wearer puts that format on, they are borrowing that same impersonal tone. The shirt is not saying I dislike you. It is saying this area is restricted. That is a much easier thing to wear in public, and a much easier thing to laugh at, because it removes the sting of rejection from the message.

We think that is the smartest thing about the design. It takes a boundary that could read as hostile and formats it as infrastructure. The crossed-out hand is not a rejection of a specific person. It is a policy. Policies are easier to accept than insults.

What the shirt is not

It is not a claim about introversion as a clinical category, and we would not read it as one. The retail title uses the word introvert, but the artwork itself does not. The artwork says not friendly and do not touch, which are behavioral instructions, not personality labels. We mention this because it is easy to collapse the two, and the collapse does the design a disservice. The shirt is funnier and more useful as a boundary marker than as a personality badge.

It is also not a substitute for actually telling people what you need. A shirt is a low-stakes signal, not a substitute for a conversation. We would not want to overstate its power. But as a low-stakes signal, it is unusually well constructed: legible, consistent, and formatted in a visual language that most people already know how to read.

Where it fits

We keep a small decision log for objects like this — things that are not tools but that shape how a team interacts. A shirt is not a runbook. But the question it raises is the same one we ask about runbooks: does the reader know what to do next? The Not Friendly Do Not Touch design answers that question clearly. The reader knows exactly what to do next. They know not to touch. They know the wearer is not friendly. The interface is complete.

If you want to look at the listing, it is here: Not Friendly Do Not Touch Funny Warning Sign Introvert. We are not going to pretend we tested it. We read the artwork, and the artwork is doing something specific and coherent. That is enough for us to recommend it as a well-designed object, which is a different thing from a recommendation to buy it.

What we’d do differently

If we were designing a shirt like this, we would probably be tempted to add more text — a second line, a clarification, a joke that undercuts the boundary. We would be wrong. The strength of this design is its restraint. The caution triangle, the crossed-out hand, the three words. Nothing else. The temptation to explain is the enemy of a usable signal, and this shirt resists it. We would do well to remember that the next time we are writing a runbook and reach for a fourth paragraph of context. Sometimes the crossed-out hand is enough.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

On a dark shirt, centered, in very small lowercase letters, the word meh. appears. No animal illustration. No pride variant. No diagram, no arrow, no punchline in a second line. The period is part of the mark. The scale is the argument: the word is set small enough that a reader has to be close to parse it, which means the shirt does not perform for a room. It performs for one person at conversational distance, if it performs at all.

That is a different object from the slogan shirt as it usually works. Most slogan shirts are legible across a parking lot. They are designed to be read by strangers, which makes them a kind of broadcast. This one is designed to be read by someone already close enough to notice. The difference matters for anyone who has spent time thinking about how teams communicate under stress, because it is the same difference between a status page and a runbook.

We keep a small decision log for our own systems, and one recurring entry is about audience. A status page is written for people who are not us: customers, support, whoever is watching the graph. A runbook is written for the person who will be paged at 3 a.m. and needs the next command, not a narrative. Both are text. Both are terse. But they are calibrated to different distances, and confusing them produces the specific failure where a status page reads like an internal note or a runbook reads like a press release.

The meh. shirt is calibrated to the close distance. It is not a status page. It is not trying to inform a crowd that something is wrong. It is closer to a runbook entry that says only what the operator needs and nothing more: a single word, a period, done. The period is doing real work. Without it, meh is a sound. With it, meh. is a verdict. The punctuation turns an interjection into a statement, and the small type keeps the statement from becoming a speech.

I want to be careful about what this observation supports. It supports a reading of the artwork. It does not support a claim that the shirt improves anything, communicates a policy, or represents a movement. It is a garment with a word on it. The interesting question is not whether it is deep. The interesting question is what kind of message survives being set this small, and why a designer would choose that constraint.

Small type is a filter. It selects for proximity and attention. In a work context, the messages that survive that filter tend to be the ones that do not need volume: a boundary, a handoff, a single line in a channel that says the thing and stops. The messages that fail the filter are the ones that need an audience to complete them. A joke that requires a laugh track is not the same object as a joke that lands in a quiet room. The meh. shirt is built for the quiet room.

There is a useful distinction here between a slogan and a usable promise. A usable promise tells you what will happen next. It has an owner, a trigger, and a condition. A slogan tells you what someone feels or claims. The meh. shirt is a slogan, and I think that is the correct category for it. It does not promise uptime, recovery, or anything operational. It does not pretend to be a runbook. It is a mood rendered as typography, and the restraint is what keeps it honest. A shirt that tried to be a usable promise would need more words, and more words would make it worse.

This is where the object connects to a habit I think technical teams should steal from designers: decide the distance before you decide the words. If the message is for the person already in the room, you can be terse. If it is for the person who has not arrived yet, you need context. Most communication failures I have watched are distance failures, not tone failures. Someone writes a close-distance message and broadcasts it, or writes a broadcast message and wonders why the on-call engineer ignores it. The fix is rarely more empathy. It is usually a smaller audience and a shorter sentence.

The shirt is available as a standard t-shirt through the listing at https://www.amazon.com/dp/B0FXC6KKSR. I am describing the artwork as observed in the native garment preview, not a wearing experience. I have not tested the fabric, and I am not going to pretend that a shirt is a reliability practice. It is an object with a word on it, and the word is set small on purpose.

What I keep returning to is the period. In a runbook, the period is the end of a step. It means the command is complete and the next thing can happen. In the shirt, the period does something similar: it closes the word so it does not trail off into a question or a shrug. meh. is not ambivalence. It is a decision to stop. That is a small, repeatable move, and it is one I would like to see more of in incident communication: say the thing, end the sentence, wait for the next signal.

What we’d do differently

If we were writing the listing copy, we would resist the urge to explain the joke. The artwork is already doing the work, and explanation would only add distance. We would also avoid claiming the shirt is for a particular personality type, because the design does not establish that. The observation is the type: small, centered, lowercase, dark ground, period included. Everything else is the reader’s business.

A bus factor audit is not a documentation project. It is a mapping exercise. The deliverable is a per-service list of people who can actually restart the service, plus the evidence behind each name. If the list has names but no evidence, you have a hypothesis, not a map.

For a team of 2–15 engineers who are their own on-call rotation, the audit has one job: find the services where the answer to “who can restart this?” is a single person, a stale runbook, or a name that has never done it under pressure. Then turn those gaps into drills, shadowing, and runbook work before the next incident finds them for you.

Why this is a mapping exercise, not a documentation project

Documentation projects tend to produce artifacts that describe how a service works. A bus factor audit produces a different artifact: a table that says who can perform a specific recovery action and how you know. The distinction matters because a runbook can be complete and still leave you with a bus factor of one. The runbook is not the capability. The person who has executed the runbook is the capability.

PagerDuty’s incident response guidance notes that knowledge of how to handle incidents is often lost when a team member leaves, which is why a documented process becomes more crucial over time. That is the failure mode the audit is designed to surface before it becomes an incident. The audit does not replace the documentation; it tells you where the documentation is the only thing standing between you and an unowned service.

NIST SP 800-34 Rev. 1 frames contingency planning as a process of evaluating information systems and operations to determine requirements and priorities. A bus factor audit is a narrow, repeatable version of that evaluation, scoped to restart and recovery actions rather than the full contingency plan.

Define the unit of audit: service, restart action, evidence

The unit of work is not the service as a whole. It is the restart action. A single service may have several restart paths, and different people may be able to perform each one:

  • Process restart on the primary host or container.
  • Failover to a standby or secondary region.
  • Restore from backup (database, object store, configuration).
  • Rebuild from infrastructure-as-code when the host is gone.
  • Break-glass access path when normal credentials are unavailable.

For each path, record the evidence type. Three levels are enough for most lean teams:

  • Has done it in an incident. The strongest evidence. The person executed the action under real pressure, with real consequences.
  • Has done it in a drill. Strong evidence. The person executed the action in a controlled setting, ideally with a time constraint and a defined success condition.
  • Has read the runbook. Weak evidence. This is a starting point, not a qualification.

Google’s SRE workbook describes a pre-on-call checklist that includes administering production jobs, rolling back a bad software push, bringing up additional serving capacity, and using monitoring systems. That checklist is a useful model for the evidence bar: the question is not “have you read about this?” but “have you practiced this?”

Build the map

One row per service. Columns for primary, backup, evidence type, last verified date, and known gaps. Keep it in a place the team already looks at, not a separate tool that will be forgotten.

Service Restart path Primary Backup Evidence Last verified Known gaps
payments-api Process restart Engineer A Engineer B Incident (2025-08) 2025-08-14 None
payments-api Restore from pgBackRest Engineer A — Read runbook only 2025-06-02 No second person has executed a restore
internal-tools Rebuild from Terraform Engineer C Engineer D Drill (2025-09) 2025-09-20 Break-glass path untested

The table is deliberately small. For a 2–15 engineer team, the audit should be completable in a few sessions. If it is taking longer, the scope is too broad. Start with the services that have the highest blast radius and the fewest people.

Use the map to find single points of failure

Three patterns are worth flagging explicitly:

  • One name, no backup. The service has a primary but no second person who has executed the restart path. This is the classic bus factor of one.
  • Names but no evidence. The service has a primary and a backup, but both are at “has read the runbook.” This is a bus factor of one with extra steps.
  • The runbook is the only artifact. No one on the current team has executed the restart path, and the runbook was written by someone who has left. This is the highest-risk category because the runbook may be stale in ways no one can see.

Google’s SRE workbook notes that playbook details go out of date at the same rate as production environment changes, and that for daily releases, playbooks might need an update on any given day. A runbook that has not been executed by a current team member is a runbook whose staleness is unknown.

Turn gaps into drills

The audit’s output should feed directly into drills and shadowing, not into a separate documentation backlog. The goal is to move people from “has read the runbook” to “has done it in a drill.”

Google’s SRE workbook describes the Wheel of Misfortune as a training exercise where teams role-play recent incidents to practice debugging production problems. A bus factor drill is a narrower version: pick a service with a gap, pick a person who has not executed the restart path, and run the restart in a controlled setting. The success condition is not “the service came back” but “the person can describe what they did and why.”

For restore paths, a restore lottery is a useful format. One person is selected at random, given the backup and the runbook, and asked to restore to a point in time. The exercise surfaces both the person’s capability and the runbook’s accuracy. If the restore fails because the runbook is wrong, that is a finding, not a failure.

For services with no second person, shadowing is the first step. The primary executes the restart path while the backup observes and asks questions. The next step is the reverse: the backup executes while the primary observes. Only after that does the backup count as evidence.

Keep the map alive

A bus factor map that is not reviewed becomes a stale spreadsheet. Tie the review to artifacts the team already uses:

  • On-call handoffs. When the rotation changes, check whether the new on-call has executed the restart paths for the services they are now responsible for.
  • Playbook updates. When a runbook changes, mark the corresponding map row as unverified until someone executes the new path.
  • Postmortem action items. When a postmortem identifies a recovery gap, add it to the map with a review date.

The Google SRE book states that an unreviewed postmortem might as well never have existed, and that regular review sessions are encouraged to ensure each completed draft is reviewed. The same logic applies to the bus factor map: an unreviewed map is a map you cannot trust.

Google’s SRE book lists common postmortem triggers including user-visible downtime or degradation beyond a certain threshold, data loss of any kind, on-call engineer intervention, a resolution time above some threshold, and a monitoring failure. Any of these triggers is also a signal to re-check the map for the affected service.

What to do when the map shows a gap you cannot close this quarter

Not every gap can be closed immediately. A team of five engineers cannot train everyone on everything in a single quarter. When a gap cannot be closed, record it explicitly:

  • The service and restart path.
  • The current evidence level.
  • The accepted risk, in plain language.
  • A review date.

The alternative is to leave the service silently unowned, which is the condition the audit was meant to find. An accepted risk with a review date is a decision. An unrecorded gap is an accident waiting for a trigger.

For teams running on AWS, GCP, or bare metal, the audit is the same exercise. The AWS Well-Architected Framework reliability pillar notes that achieving reliability can be challenging in traditional on-premises environments due to single points of failure, lack of automation, and lack of elasticity. The bus factor audit is a way to find the human single points of failure that architecture diagrams do not show.

A short, repeatable audit cadence

For a 2–15 engineer team, a quarterly cadence is enough to keep the map current without turning it into a project. The cadence can be as simple as:

  1. Pick the three services with the highest blast radius.
  2. For each, list the restart paths and the people who can execute them.
  3. Mark the evidence level for each person.
  4. Identify one gap per service and schedule a drill or shadowing session.
  5. Update the map with the drill result and a new review date.

The audit is not a one-time report. It is a working artifact that changes as the team changes. The goal is not to eliminate every gap. The goal is to know where the gaps are, so that when the pager fires at 03:00, the person who answers has a realistic path to recovery.

Frequently asked questions

How is a bus factor audit different from a runbook review?

A runbook review checks whether the documentation is accurate. A bus factor audit checks whether a person can execute the documented steps. A runbook can pass review and still leave you with a bus factor of one if no one on the current team has executed it.

How many people need to be able to restart a service?

There is no universal number. The audit’s job is to make the current state visible, not to enforce a threshold. For a team of 2–15 engineers, the practical question is whether the service has at least one person who can execute the restart path when the primary is unavailable, and whether that person has evidence beyond reading the runbook.

What if the only person who can restart a service is leaving?

That is the highest-priority gap the audit can find. The response is a drill or shadowing session before the person leaves, with the goal of moving at least one other person to “has done it in a drill.” If that is not possible, the gap should be recorded as an accepted risk with a review date and a plan for the next quarter.

Should the audit include break-glass access?

Yes. Break-glass access is a restart path. If the normal credentials are unavailable, the break-glass path is the one that matters. The audit should record who can use it and whether they have tested it.

How often should the map be reviewed?

Quarterly is a reasonable default for a lean team. The map should also be reviewed after any incident that required a restart, after any on-call rotation change, and after any playbook update that changes a restart path.

For a related practice, see Write the Recovery Checklist Before You Need It, which covers the checklist side of the same problem: what to have ready before the restart is needed.

Most handover notes fail the same way. The author writes a careful description of the backup system — where the repository lives, which tool runs nightly, roughly how a restore works — and the reader, three days into a vacation, discovers the note describes intent rather than a path. The repository is reachable. The command is almost right. The backup set the reader picked cannot actually be restored to the target they need, and nothing in the note said so.

The unit of value in a handover note is not the description of the backup system. It is the executable recovery path: the exact command, the inputs it needs, and the observable that proves it worked. If the substitute cannot run it, the note has failed regardless of how complete it reads.

This is a craft note for lean teams — two to fifteen engineers, no dedicated SRE department, one on-call rotation — about what to write down before the person who can restore the database becomes unreachable for two weeks. It is not a template to fill in. It is a set of decisions about what belongs in the note and what does not.

Start from the failure mode, not the format

The failure mode is specific. The author leaves. A restore is needed. The substitute opens the note, finds a paragraph explaining that PostgreSQL maintains a write ahead log in pg_wal/ that records every change to the database’s data files, and finds a command that looks plausible. They run it. It completes. They start PostgreSQL. It comes up. The table they needed is still missing, because the backup they restored ended after the data was lost, and PostgreSQL can only play forward, not backward.

That is not a hypothetical. The pgBackRest user guide documents exactly this scenario in its point-in-time recovery walkthrough: if the wrong backup is selected, recovery to the required time target fails, and the log output does not make the failure obvious. The key is to look for the presence of the “recovery stopping before…” and “last completed transaction…” log messages. If they are not present, the recovery to the specified point-in-time was not successful.

A note that does not tell the reader to check for those log lines has not equipped them to know whether they succeeded. A note that does not tell them which backup sets are valid together has not equipped them to choose.

What the note must contain for each recovery scenario

For each scenario the team actually cares about — full cluster restore, point-in-time recovery to a timestamp, single-database restore, file-level restore from the object store — the note needs four things:

  1. The exact command, with the flags that matter, not a paraphrase.
  2. The inputs the command needs: which repository, which backup set, which target, which key material, which host.
  3. The dependency chain: what else must be valid for this command to produce a usable result.
  4. The observable that proves it worked — a log line, a query result, a checksum, a row count.

Anything else is context. Context is useful, but it is not the note. The note is the four items above, written so the reader’s first hour is spent executing rather than interpreting.

The PostgreSQL and PITR path

Continuous archiving combines a file-system-level backup with backup of the WAL files. Recovery restores the file system backup and then replays the backed-up WAL files to bring the system to a current state. That is the mechanism. The note’s job is to make the mechanism’s prerequisites explicit.

The first prerequisite is a continuous sequence of archived WAL files extending back at least as far as the start time of the backup. Not “some WAL files.” A continuous sequence, back to the backup’s start. If there is a gap — an archiving failure that was not noticed, a repository that filled, a version mismatch that stopped archiving — the backup is not recoverable to a point after the gap. The note should say where to check for gaps and what a gap looks like.

The second prerequisite is that pg_dump and pg_dumpall are not part of this path. They do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. If the team has both a logical dump and a physical backup, the note should say which one is for which purpose, because a substitute under pressure will reach for the one they recognize.

The third is that WAL archiving does not restore changes made to configuration files — postgresql.conf, pg_hba.conf, pg_ident.conf. Those are edited manually rather than through SQL operations. The note should say where the current configuration lives and how it is restored, because a cluster that comes up with the wrong pg_hba.conf is a cluster that comes up but does not accept the connections the application needs.

The fourth is the disk-space failure mode. If the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until space is freed. The note should say which filesystem to check and what the safe threshold is, because the substitute may be the person who has to free space at 2 a.m.

The pgBackRest path

pgBackRest’s restore command defaults to selecting the latest backup from the first repository where backups exist. That default is convenient and dangerous. The note should state it, because a reader who assumes the tool will pick the right backup will not pass --set or --repo, and the default may not be the backup they need.

For point-in-time recovery of --type=time or --type=lsn, pgBackRest selects a backup whose stop time is before the target time, or whose stop LSN is less than or equal to the target LSN. If no matching backup is found, it raises an error for both types; it does not fall back to the latest backup. That behavior belongs in the note verbatim, because the failure mode is a reader who assumes a fallback exists and does not check the error.

The dependency chain is the other thing the note must state. A differential backup requires both the differential backup and the previous full backup to be valid in order to restore. An incremental backup requires all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup to all be valid. The note should name which sets are valid together for the scenarios the team cares about, rather than pointing at “the backup.”

Two operational facts belong in the note as well. pgBackRest encrypts the repository based on a user-provided password, so the password is part of the recovery path, not a separate concern. And the local and remote pgBackRest versions must match exactly; if there is a mismatch, WAL archiving and backups will not function until the versions match. A note that does not record the version in use at the time of writing will be wrong after the next upgrade.

The restic and age path

For file-level backups, the note’s job is to separate two claims that are easy to conflate: “the repository exists” and “this person can decrypt and restore it.” Only the second one matters during a vacation.

age is a file encryption tool, format, and Go library with small explicit keys, post-quantum support, no config options, and UNIX-style composability. Files can be encrypted to multiple recipients by repeating the recipient flag, and every recipient will be able to decrypt the file. Identity files contain one or more secret keys, one per line, or an SSH key.

The note should record: where the repository lives, which recipients the backups are encrypted to, where the identity files are stored, and how the substitute obtains access to them. If the identity file is itself passphrase-protected and stored remotely, the note should say so, because that is a second access step that a substitute will not discover until they need it.

The note should also record what a successful restore looks like. For a restic repository, that is a restic check result and a test restore of a known file. For an age-encrypted archive, that is a decryption that produces the expected file. The observable is the point.

Break-glass access and offboarding hygiene

The handover note is where access hygiene and recovery access meet, and the tension is real. Least privilege says the substitute should not hold standing credentials they do not need. Recovery says the substitute must be able to reach the repository and the key material when the author is unreachable.

The resolution is to write down the break-glass path rather than to grant standing access. The note should name: the mechanism by which the substitute obtains elevated access, who approves it, how long it lasts, and how it is revoked. It should name the offboarding step that removes the author’s access when they return, so the break-glass grant does not become permanent. And it should name the person who can approve the grant when the author is the one who normally would.

This is the part of the note most likely to be skipped, because it is administrative rather than technical. It is also the part most likely to be needed, because a substitute who cannot reach the repository cannot run any of the commands the rest of the note describes.

The pre-vacation drill

The note’s only real test is running it as the substitute would. Not reading it. Running it.

The author picks a scratch target — a throwaway instance, a staging cluster, a container — and executes the note’s commands in the order the note presents them, using only the note as a guide. Every place the author has to supply knowledge that is not in the note is a gap. Every command that fails because a prerequisite was not stated is a gap. Every observable the author checks without the note saying to check it is a gap.

The record of that drill is the note’s revision history. It is more useful than a version number, because it says what was tested and when. A note that was drilled against pgBackRest 2.59.3 and PostgreSQL 17 is a different artifact from a note that was written against an earlier version and never re-run.

The drill also surfaces the failure behavior the note needs to record. A substitute who has run the commands once, on a scratch target, with the author watching, is a substitute who will recognize the difference between a restore that succeeded and a restore that completed.

Review, ownership, and re-verification

The note should have a named reviewer and a date. The reviewer is not a second author; they are the person who reads the note as a substitute would and reports what they could not follow. The date is the point from which re-verification is due.

The re-verification trigger is tied to change, not to a calendar. A note that was true at one pgBackRest version may be false at the next, because the local and remote versions must match exactly. A note that was true before a topology change may point at a repository that no longer exists. A note that was true before a key rotation may reference an identity file that has been replaced. The note should say what changes invalidate it, so the team knows when to re-drill.

This is the same discipline the SRE literature applies to postmortems. A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring. An unreviewed postmortem might as well never have existed. The same is true of a handover note. An unreviewed note is a document, not a recovery path.

It is worth defining, before the vacation rather than during it, what triggers a note review. The SRE book’s guidance on postmortem criteria applies here: define the criteria before the event so everyone knows when the process is necessary. For a handover note, the criteria are simpler — a version change, a topology change, a key rotation, or a drill that failed.

What the note is not

The note is not a postmortem. A postmortem explains what happened and why; a handover note explains what to do next. The two share a discipline of specificity and a refusal to blame, but they serve different readers at different times.

The note is not a runbook for every service. It is the recovery path for the things the author uniquely knows how to restore. If the team already has a runbook that a substitute can follow, the note should link to it rather than duplicate it. The Recovery Checklist Before You Need It is a useful companion for the general shape of a recovery checklist; the handover note is the specific, dated, drilled instance of it for one person’s absence.

The note is not a knowledge dump. A note that explains the history of the backup system, the reasons for each architectural choice, and the tradeoffs considered is a design document. Those are valuable. They are not what a substitute reads at 2 a.m. The note should be short enough to read in the first hour and specific enough to execute in the second.

Frequently asked questions

How long should the handover note be?

Long enough to contain the executable recovery paths for the scenarios the team cares about, and no longer. For a single PostgreSQL cluster with pgBackRest and a restic repository, that is typically a few pages: one section per scenario, each with the command, the inputs, the dependency chain, and the observable. Length is not the measure. The measure is whether a substitute can run it without asking the author a question.

Should the note include the backup password or key material?

No. The note should say where the key material is stored and how the substitute obtains access to it, not contain the material itself. pgBackRest encrypts the repository based on a user-provided password, and age identity files contain secret keys; both belong in a secrets manager or an equivalent access-controlled store, with the note recording the path and the approval step. A note that contains the key is a note that leaks when it is shared.

What if the substitute has never run a restore before?

Then the pre-vacation drill is not optional. The author runs the note’s commands as the substitute would, on a scratch target, and records what broke. The substitute should run them too, with the author available but not intervening. A note that has only been read is a hypothesis; a note that has been executed is a procedure.

How often should the note be re-verified?

Tie re-verification to change rather than to a calendar. A pgBackRest version change, a PostgreSQL major-version upgrade, a repository migration, a key rotation, or a topology change all invalidate parts of the note. The note should name those triggers so the team knows when to re-drill. A calendar-based cadence is a fallback for teams that cannot detect the changes, not the primary mechanism.

Does the note replace a postmortem when a restore fails?

No. If a restore fails during the author’s absence, that is an incident, and it merits a postmortem. The SRE book’s guidance is that postmortems are expected after any significant undesirable event, and that a blameless postmortem focuses on identifying the contributing causes without indicting any individual or team. The handover note is what the team uses during the incident; the postmortem is what the team uses afterward to make the next note better.

What is the single most important thing to write down?

The observable that proves the restore worked. Commands can be copied from documentation. The dependency chain can be reconstructed from the tool’s behavior. But the observable — the log line, the query result, the checksum — is the thing that tells the substitute whether they are done or whether they have produced a successful-looking but useless result. Write that down first, and write it for every scenario.