The failure you need to catch is the run that never happened

Most backup monitoring watches for errors. That is the wrong shape for a cron job. The failure modes that hurt a lean team are quieter: the crontab was overwritten by a config-management run, the cron daemon is not running in the rebuilt container, the host was powered off at 02:30, or the job started and hung on a stale NFS mount. None of those reliably produce an error you will see. They produce silence.

A dead man’s switch inverts the reporting direction. The job reports success when it finishes; the absence of that report is the alert. Healthchecks.io describes the mechanism plainly: it listens for HTTP pings from cron jobs and scheduled tasks, stays silent while pings arrive on time, and raises an alert as soon as a ping does not arrive on time. The same documentation calls this a dead man’s switch for processes that need to run on a regular, known schedule, and lists filesystem backups and database backups among the jobs it is built to monitor (Healthchecks.io documentation).

The argument here is narrower than “add monitoring.” For a team of two to fifteen engineers who are their own on-call rotation, the cheapest useful switch is not a new daemon, dashboard, or service to operate. It is one success-only HTTP call appended to the backup job you already run, plus a grace window sized to that job’s real behavior. The rest of this article is about wiring that call correctly, sizing the window so it does not page on normal runs, and being honest about what the switch does not prove.

Why cron’s built-in mail is not the switch

Cron does have a notification path. The cron manual states that job output is mailed to the owner of the crontab, or to the user named in MAILTO, and that output can also be sent to syslog with the -s option (cron(8)). That is useful for diagnosing a job that ran and printed an error. It is not a dead man’s switch.

Healthchecks.io’s cron guide lists the practical drawbacks: MAILTO requires a configured MTA on the host, it does not notify you when the whole machine is powered off or has lost network connectivity, and routine stdout produces mail on every run, which trains people to ignore it (How to Monitor Cron Jobs). Those are drawbacks of the channel, not proof that every server lacks an MTA. The point is structural: mail from the host cannot report the host being gone.

A success ping sent to an external endpoint covers the cases mail cannot. If the host is off, no ping arrives. If cron is stopped, no ping arrives. If the crontab was wiped, no ping arrives. If the job hangs, no success ping arrives. All four look identical from outside, which is exactly what you want from a switch.

The minimal wiring: one gated call

Healthchecks.io’s guide shows the pattern as a single crontab line, with the ping chained using && so it runs only when the backup exits zero:

# run backup.sh, then send a success signal
8 6 * * * /home/me/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/your-uuid-here

The guide explains the curl flags: retry failed requests, cap the maximum execution time, and silence output unless there is an error. It also notes that the ping can instead be the final line inside the backup script, which keeps the crontab short (Healthchecks.io cron guide).

For anything with more than one step, put the ping at the end of the script under set -euo pipefail. That way any failing step aborts before the ping is reached, and the ping asserts on the whole pipeline rather than the exit code of a wrapper. A backup that runs, writes nothing, and exits zero is still a failed backup; put whatever verification the job can afford before the ping, and the switch covers that too.

Two operational details matter more than they look. First, cron runs with a minimal PATH, so use the absolute path to curl. Second, treat the ping URL as a secret. Healthchecks.io’s documentation is explicit: check UUIDs and project ping keys should be treated as secrets, because anyone who has them can send telemetry signals to your checks and disturb your monitoring (Healthchecks.io documentation). Keep the URL out of public repositories and pass it through an environment variable or a secrets store.

Sizing the grace window from observed behavior

A heartbeat check has a schedule and a grace time. Healthchecks.io defines grace time as the additional wait before alerting when a check is late, intended to absorb minor expected deviations in job execution (Healthchecks.io documentation). Its cron guide adds the practical rule: set grace time above the expected duration of the job (Healthchecks.io cron guide).

For a nightly backup, that means the window should be built from three numbers you can observe: how long the job usually takes, how long the slowest recent run took, and how much schedule drift you tolerate. A round number like “one hour” is a guess. A window derived from the p95 duration of the last thirty runs plus a margin is a decision you can defend in a postmortem.

The tradeoff is symmetric. Too tight a window pages on a normal slow night and erodes trust in the alert. Too loose a window means a skipped run is not noticed until most of the next day has passed. For a daily backup, a window that fires within a few hours of the expected completion is usually the right compromise; for an hourly job, the window has to be much shorter or the switch is decorative.

If the team is migrating to systemd timers, the same sizing applies, with two extra sources of drift to account for. The systemd.timer documentation states that AccuracySec defaults to one minute, so timers do not necessarily expire at the precise configured time, and that RandomizedDelaySec adds a random delay between zero and the configured value to spread load (systemd.timer(5)). If you use either setting, the grace window must absorb it.

Notification channels and redundancy

A switch that alerts into a channel nobody watches is not a control. Healthchecks.io supports multiple integrations, including email, webhooks, SMS, Slack, and PagerDuty, and allows more than one per check (Healthchecks.io documentation). Its cron guide recommends setting up multiple notification methods for redundancy and using different channels depending on job priority (Healthchecks.io cron guide).

For a lean team, the useful configuration is usually two channels with different failure modes: one that reaches the on-call rotation (PagerDuty or an equivalent escalation path) and one that leaves a durable record in a team channel. Email alone is a single point of failure; a chat message alone is easy to miss overnight. The switch should page a human who can act, not a channel that is read the next morning.

The runbook attached to that page should name the first three checks before anyone touches the backup repository: is the host reachable, is the scheduler running, and does the job log show a start for the expected run. Those three questions separate “the backup failed” from “the backup never started,” and they are the difference between a five-minute fix and an hour of guessing.

What the switch does not prove

A success ping proves the job reached the end of its script and exited zero. It does not prove the backup is restorable. That gap is not hypothetical: a backup can upload zero bytes, write an incomplete snapshot, or archive WAL files that cannot be replayed, and still exit zero and ping.

PostgreSQL’s documentation is direct about the WAL side. Recovery using continuous archiving requires a continuous sequence of archived WAL files extending back at least to the start time of the backup, and the documentation advises setting up and testing the archiving procedure before taking the first base backup (PostgreSQL: Continuous Archiving and PITR). It also states that the archive command must return zero exit status if and only if it succeeds, and that a nonzero status causes PostgreSQL to retry periodically. A heartbeat on the backup job does not replace monitoring the archiver itself; the same documentation advises monitoring the archiving process to ensure it is working as intended.

The practical consequence is that the ping belongs after whatever verification the job can afford, and a periodic restore drill remains necessary. The switch tells you the job ran. The drill tells you the backup is worth having. Those are different controls, and a lean team needs both. For a starting point on the drill side, see Write the Recovery Checklist Before You Need It.

Optional hardening: start and fail signals

The success-only ping is the minimum. Healthchecks.io’s Pinging API also supports start, failure, and log signals, and supports reporting a script’s exit status where zero is success and any other value is failure (Pinging API). Sending a start signal is optional, but it enables measuring job execution time and detecting a job that runs longer than its configured grace time.

That last capability is worth the extra call for a backup that can hang. A start ping arms overrun detection: if the matching success does not arrive inside the grace window, the check goes down while the job is still wedged, rather than waiting for the next scheduled run to reveal the problem. The cost is one more HTTP call and a slightly more complex script. For a nightly job that occasionally stalls on a network mount, it is usually worth it.

If the team is on systemd timers, the same pattern moves cleanly. A oneshot service can use ExecStartPost for the success ping, which runs only after ExecStart has completed successfully, and an OnFailure unit for the fast-failure signal. The heartbeat still catches the timer that never fired, which is the case OnFailure cannot see. The systemd.timer documentation also notes that Persistent=true stores the last trigger time and runs a missed activation at the next opportunity, which is useful for backups but does not remove the need for the switch: a host that never comes back still produces no ping (systemd.timer(5)).

Test the switch itself

An untested switch is a belief, not a control. The failure path has to be exercised once, deliberately, before you rely on it. The procedure is short: make the backup script exit nonzero on a test run, confirm that no success ping is sent, and confirm that the alert arrives on the expected channel within the expected window. Then restore the script and confirm the next successful run clears the alert.

Two details are easy to get wrong in that test. First, a timezone mismatch between the host and the check configuration can make pings arrive an hour early or late; Healthchecks.io’s guide names this as the likely cause when pings are consistently offset (Healthchecks.io cron guide). Second, the alert window is the schedule plus the grace time, so a test that waits only for the schedule will conclude too early that the switch is broken.

After the test, write down three things in the runbook: who receives the alert, what the first three checks are, and how to pause the check during a known maintenance window. Healthchecks.io supports a paused state for exactly this purpose, so a planned outage does not generate a false page (Healthchecks.io documentation).

Frequently asked questions

Do I need a separate monitoring service for this?

No. The pattern is one HTTP call from the job to an endpoint that expects it on a schedule. That endpoint can be a hosted heartbeat service or a self-hosted equivalent. The decision is about who operates the endpoint, not about whether the pattern works. For a lean team, the tradeoff is usually between a small external dependency and the operational cost of running another service yourself.

Should the ping fire at the start or the end of the job?

The end, and only on success. A start ping proves the scheduler fired, which is the easier half of the problem. A success ping proves the work completed. If you want overrun detection as well, send both: a start signal to arm the timer and a success signal to clear it.

What if the monitoring endpoint itself is down?

That is a real dependency, and it should be named in the runbook rather than assumed away. The practical mitigations are to use a service with its own redundancy and to keep a second, independent signal for the highest-priority job, such as a daily summary that a human reads. The switch reduces the chance of silent failure; it does not eliminate every dependency.

Does this replace restore testing?

No. The switch proves the job ran and exited zero. Restore testing proves the backup is usable. A backup that exits zero and pings can still be unrestorable, so the ping should sit after any verification step the job can afford, and a periodic restore drill remains necessary.

How do I keep the ping URL out of logs and repositories?

Treat it as a secret. Pass it through an environment variable or a secrets store, keep it out of public repositories, and avoid printing it in job output. Healthchecks.io’s documentation is explicit that public ping URLs let anyone send telemetry signals to your checks and disturb your monitoring (Healthchecks.io documentation).

The wrong default: remove the account first

In a two-to-fifteen engineer team, offboarding usually starts with the obvious action: disable the account. That is the wrong first move when the departing person’s identity is the only thing holding a pipeline together. The account is a human object. The pipeline depends on machine objects — deploy keys, CI/CD-scoped organization roles, cross-account trust policies, service-account impersonation grants — and several of those survive account removal entirely.

GitHub’s own repository-role documentation is explicit about one of them: when someone adds a deploy key to a repository, any user who has the private key can read from or write to the repository, depending on the key settings, even if they are later removed from the organization. Account removal is not credential removal. The offboarding sequence has to follow dependency direction, not calendar order.

This is a procedural argument, not a vendor recommendation. GitHub, GitLab, AWS, and Google Cloud document the mechanisms; none of them publish an offboarding order. The order below is a recommendation for lean teams, built from those mechanisms.

Step 0 — Inventory what the person’s identity is propping up

Before touching access, enumerate the objects that name the departing person or their personal credentials. In practice that means four lists:

• Deploy keys and personal tokens attached to repositories the pipeline deploys from or to.
• CI/CD-scoped organization roles. GitHub offers a predefined CI/CD admin role that grants admin access to manage Actions policies, runners, runner groups, hosted compute network configurations, secrets, variables, and usage metrics for an organization. If the departing person holds it, the pipeline’s secret store is in scope.
• Cross-account trust policies. An AWS cross-account IAM role includes a trust policy that allows principals in another account to assume it. If the trust policy names the person’s IAM user or a role only they can assume, revocation breaks the pipeline’s path into the target account.
• Service-account impersonation grants. In Google Cloud, a principal granted a role that allows service account impersonation can access everything the service account can access. The grant is on the person; the blast radius is the pipeline.

Write the list down. This inventory is the artifact that makes the rest of the sequence reversible.

Step 1 — Re-home machine credentials before revoking anything

Replace personal deploy keys and personal tokens with team-owned or pipeline-owned credentials. The replacement should be a key or token whose lifecycle is tied to the pipeline, not to a human account. Then confirm the pipeline runs green on the replacement before the original is revoked.

Two platform details matter here. First, GitHub’s deploy-key warning cuts both ways: a new team-owned deploy key is only as safe as its private-key custody, so store it where the pipeline reads it and nowhere else. Second, GitLab’s membership model distinguishes direct from inherited members. A direct project member can be removed from the project directly, but an inherited member from a parent group can only be removed from the parent group itself. If the departing person’s pipeline access comes through a group, re-homing means changing the group, not the project.

For a worked example, consider a hypothetical two-person team where the departing engineer’s personal SSH deploy key is the only write path into the deploy repository. The correct sequence is: generate a new deploy key owned by the pipeline, add it to the repository, run a deploy, confirm the deploy succeeded, then remove the old key. Removing the old key first would orphan the pipeline at the exact moment the team has the least capacity to fix it.

Step 2 — Re-point trust and impersonation

Trust policies and impersonation grants are the second class of object that outlives an account. AWS recommends scoping role trust to specific principals rather than :root, because referencing :root may allow more principals to assume the role than intended. That recommendation is a security control, but it also creates an offboarding dependency: if the trust policy names the departing person’s role, the pipeline loses its path the moment that role is deleted.

The fix is to re-point the trust policy at a role or group that will outlive the person — a pipeline service role, a team role, or a group whose membership is managed separately. AWS CodePipeline’s cross-account documentation describes the shape of this: a cross-account role with a trust relationship policy that allows the pipeline’s service role in one account to assume a role in the other. The trust names the service role, not a human.

Google Cloud has the same shape. If a user holds roles/iam.serviceAccountTokenCreator on a service account the pipeline uses, revoking the user’s access is correct, but only after the pipeline’s own identity has been granted the impersonation it needs. Google Cloud also warns against deleting service accounts that are still used by running instances. The same caution applies to any service account a pipeline depends on: delete the human grant, not the machine identity.

Step 3 — Downgrade human access before removing it

Handover work usually needs the departing person to answer questions, review a runbook, or confirm a credential location. Removing all access on day one makes that handover harder than it needs to be. The middle step is a downgrade: move the person from privileged roles to the least role that still lets handover proceed, then remove.

GitHub’s organization-role documentation is useful here. Organization owners have complete administrative access, and GitHub recommends limiting the role but keeping it assigned to no fewer than two people. If the departing person is one of two owners, the downgrade step is also an availability step: promote the second owner before demoting the first. A two-person team with a single owner is a structural single point of failure, and offboarding is the moment that failure becomes visible.

GitLab’s access-expiration feature offers a lighter version of the same control: a direct project member can be given an access expiration date, after which they can no longer access the project. That is a reasonable interim state for a handover window, provided the expiration is actually enforced and not extended indefinitely.

Step 4 — Revoke, then verify for surviving credentials

Only after the pipeline runs on team-owned credentials, trust policies name non-human principals, and the second owner exists should the account be removed. Then verify. Account removal does not touch several classes of object:

• Deploy keys remain valid for anyone holding the private key, per GitHub’s documentation.
• Forks and webhooks survive membership removal. GitLab advises verifying that a removed member has not forked the private repository or created webhooks, because existing forks continue to receive changes from the upstream project and webhooks continue to receive updates.
• Service-account keys are separate credentials. Google Cloud treats service account keys as a security risk if not managed correctly and recommends more secure alternatives where possible; if a key exists, its existence is independent of the human account that created it.

Verification is a checklist, not a feeling. For each object on the Step 0 inventory, confirm the object is either gone or re-homed. Then run a deploy and a rollback. A pipeline that deploys but cannot roll back is not verified.

Step 5 — Record the sequence as a runbook

The value of this sequence is that the next offboarding is a checklist rather than an archaeology project. Write it down in the same place the team keeps its recovery procedures. The Recovery Checklist Before You Need It is a useful model for the format: ordered steps, named owners, and a verification gate at each stage.

The runbook should record the dependency order explicitly — machine credentials first, trust and impersonation second, human access third, account removal last — and it should name the platform-specific objects that survive removal. That last part is what turns a generic offboarding checklist into something a lean team can actually execute at 6 p.m. on a Friday.

Why this order, and not another

The order follows dependency direction. A pipeline depends on machine credentials and trust relationships; a human account depends on nothing the pipeline needs. Revoking in the opposite order — human first, machine second — creates a window in which the pipeline is broken and the person who could explain why is already locked out.

The tradeoff is time. This sequence takes longer than disabling an account, and in a security incident the calculus changes: if the departure is adversarial, the human account comes first and the pipeline is treated as potentially compromised. For a routine departure, the dependency order is the safer default because it keeps production running while access is unwound.

Neither order is free. The point is to choose deliberately, write down which order applies to which situation, and rehearse the routine one before it is needed.

FAQ

Does removing someone from the GitHub organization revoke their deploy keys?

No. GitHub’s documentation states that anyone holding a deploy key’s private key can read from or write to the repository, even if they are later removed from the organization. Deploy keys must be enumerated and removed separately.

What happens to a GitLab member who was added through a parent group?

An inherited member can only be removed from the parent group itself, not from the project. Removing them from the project view will not work if their membership is inherited.

Can an AWS cross-account trust policy break when a person leaves?

Yes, if the trust policy names the person’s IAM user or a role only they can assume. AWS recommends scoping trust to specific principals rather than :root, which makes the named principal an offboarding dependency. Re-point the trust at a pipeline service role before removing the human principal.

Should we delete the service account a pipeline uses when the person who created it leaves?

No. Google Cloud warns against deleting service accounts that are still used by running instances, and the same logic applies to any service account a pipeline depends on. Revoke the human’s impersonation grant, not the machine identity.

How many organization owners should a small team have?

GitHub recommends limiting the owner role but keeping it assigned to no fewer than two people. For a two-person team, that means the second owner must be promoted before the first is demoted.

The password change is the easy part

Rotating a root database password is a one-line SQL statement. The risk sits around it: connections drop, a standby reconnects with the wrong credential, and a replication slot stops advancing. On a lean team with no dedicated SRE, that slot can quietly pin WAL in pg_wal until the disk fills. PostgreSQL’s own documentation is blunt about the mechanism: “A slow or failed standby server that uses a replication slot will have the same effect” as failed WAL archiving — old WAL files accumulate until the situation is resolved (PostgreSQL 18, WAL Configuration).

This is a runbook for treating credential rotation as a replication-slot safety problem. The rotation itself is trivial. The controls that matter are a slot baseline before the change, a slot confirmation after it, and a rollback path you have actually rehearsed.

Why a password change can touch slot state at all

Two separate credentials are usually in play, and conflating them is the first mistake:

• The application role password, changed with ALTER ROLE ... PASSWORD. Existing authenticated sessions are not terminated by the statement; new connections must present the new password.
• The replication connection credential, carried in the standby’s primary_conninfo or in a ~/.pgpass file on the standby. This is a different role and a different connection path.

Rotating one does not rotate the other. If you change the replication role’s password on the primary and do not update the standby’s connection string, the WAL receiver cannot authenticate. The standby stops streaming. If a physical replication slot is in use, that slot stops advancing, and WAL retention follows the slot’s restart_lsn.

The PostgreSQL documentation notes that “abrupt disconnection of a streaming client might leave an orphaned connection slot behind until a timeout is reached,” and recommends setting max_wal_senders slightly above the expected client count so disconnected clients can reconnect immediately (PostgreSQL 18, Replication). That is the documented failure shape: a connection that goes away and does not come back promptly.

Pre-flight: record the slot baseline

Before touching any credential, capture the state you will compare against. The pg_replication_slots view is the source of truth for slot identity, activity, and WAL availability. The documentation states that WAL availability of replication slots is visible there (PostgreSQL 18, Replication).

SELECT slot_name,
       slot_type,
       active,
       active_pid,
       restart_lsn,
       confirmed_flush_lsn,
       wal_status,
       inactive_since
FROM pg_replication_slots
ORDER BY slot_name;

Record the output in the change ticket or decision record. Also capture the current WAL position and the size of pg_wal:

SELECT pg_current_wal_lsn();

And on the host:

du -sh $PGDATA/pg_wal

Two checks matter more than the rest. First, is the slot active? An inactive physical slot is not necessarily broken, but it is the state in which WAL retention becomes a liability. Second, what is wal_status? If it is already lost or unreserved, the rotation is not your first problem.

Note the default retention behavior. max_slot_wal_keep_size defaults to -1, which means “replication slots may retain an unlimited amount of WAL files” (PostgreSQL 18, Replication). If you have not set a bound, a stalled slot has no ceiling. That is a configuration decision to revisit separately, but it changes how much urgency the post-change check deserves.

Order of operations

The sequence below keeps the old credential valid until the new one is proven. The window is short and logged; that is a rollback property, not a security compromise.

1. Create the new credential path first. For the replication role, set the new password on the primary. For the application role, set the new password and update the application’s secret store. Do not retire the old value yet.
2. Update the standby’s connection string. Edit primary_conninfo (or the .pgpass entry) on the standby. The documentation states that if primary_conninfo is changed while the WAL receiver is running, that process is signaled to shut down and expected to restart with the new setting (PostgreSQL 18, Replication). Plan for a brief interruption in streaming.
3. Confirm the standby reconnects. Check pg_stat_replication on the primary for a streaming state, and check the standby’s log for authentication errors.
4. Confirm the slot is advancing. Re-run the pg_replication_slots query. restart_lsn should move forward, and active should be true for the physical slot used by the standby.
5. Retire the old credential. Only after steps 3 and 4 pass. Record the time of retirement.

For the application role, the equivalent check is a new connection using the new password, followed by a normal query. Existing sessions continue until they disconnect; that is expected and not a signal of failure.

Two checks, not one

“Replication is streaming” and “the slot is advancing” are different observations. A connection can appear healthy in pg_stat_replication while a slot’s restart_lsn is stale — for example, if the standby is connected but not acknowledging WAL, or if a different slot is the one holding retention. The WAL accumulation is the thing that bites a lean team, and it is visible in pg_replication_slots, not in the connection state alone.

After the change, compare the new restart_lsn to the baseline. If it has not moved, treat that as an incident, not a cosmetic warning. The relevant question is not “is the standby connected?” but “is the slot that governs WAL retention still moving forward?”

The orphaned-slot trap after failover or switchover

Password rotation is not the only way to end up with a slot that does not advance. A slot exists solely on the instance that created it; PostgreSQL does not replicate it to standbys, so after a failover or switchover the new primary does not contain the old primary’s slot (CloudNativePG 1.18, Replication). Operators such as CloudNativePG manage physical HA slots to fill that gap, but logical slots and slot synchronization have their own failure modes.

A reported CloudNativePG issue documents one: with PostgreSQL 17’s slot synchronization, a demoted primary can retain a locally created slot marked synced=false, and “PostgreSQL’s slot sync worker refuses to overwrite slots that aren’t marked as synced=true” (CloudNativePG issue #9969). The same report notes that the worker aborts the whole cycle on the first name clash, so one leftover slot can block synchronization of all slots. The observed effect in that report was WAL accumulation on the replica.

This matters for rotation because the same verification query catches both problems. If you rotate a credential and then discover that a slot is inactive or that wal_status has degraded, you need to know whether the cause is the credential change or a pre-existing slot-identity problem. The baseline from the pre-flight step is what lets you tell the difference.

Handling the password itself

The PostgreSQL documentation cautions that an unencrypted password passed to ALTER ROLE “will be transmitted to the server in cleartext, and it might also be logged in the client’s command history or the server log,” and points to psql’s \password command as a way to change a role’s password without exposing the cleartext password (PostgreSQL 18, ALTER ROLE). For a root or replication role, use the interactive path or a secret-management tool that does not put the value in shell history.

This is a hygiene point, not a slot point. It belongs in the runbook because the rotation is the moment the credential is most likely to leak into a terminal log or a CI transcript.

Rollback drill

The rollback is: restore the previous credential on the primary, restore the previous primary_conninfo on the standby, and confirm the slot resumes advancing. Rehearse this on a standby or a staging cluster before you need it in production. The rehearsal is where you find out whether your deployment tolerates the connection drop, and whether your slot synchronization configuration recovers cleanly.

Write the exact commands into the runbook, including the baseline query and the confirmation query. The next on-call engineer should not have to reconstruct the reasoning from a chat thread. If your team keeps a recovery checklist, the rotation runbook should reference it rather than duplicate it; see Write the Recovery Checklist Before You Need It for the structure.

What to log and alert on

Three signals cover the failure modes described above:

• Slot lag growth. Track the difference between pg_current_wal_lsn() and each slot’s restart_lsn. A slot that stops advancing while the primary keeps writing is the leading indicator of WAL accumulation.
• Inactive slots. Alert on active = false for physical slots that should be streaming. The inactive_since column gives you a timestamp to reason about.
• pg_wal directory growth. A host-level check on the directory size catches the case where slot metrics are missing or misconfigured.

PostgreSQL 18 adds idle_replication_slot_timeout, which invalidates slots that have remained inactive longer than the configured duration; the default of zero disables the mechanism (PostgreSQL 18, Replication). If you run 18 and choose to enable it, understand that invalidation is a deliberate loss of the slot’s ability to resume — it is a bound on WAL retention, not a repair. For most lean teams, alerting on inactivity is the safer first step.

What goes in the decision record

The rotation runbook is a decision record as much as a procedure. Record the baseline query output, the exact commands used, the time the old credential was retired, and the post-change slot state. If the rotation is routine, the record is short. If it goes wrong, the record is the difference between a blameless postmortem and a guessing game.

The argument in one sentence: the password change is not the risk; the connection churn and the slot-state verification around it are. Sequence the runbook so the credential change sits between a slot baseline and a slot confirmation, and keep the old credential valid until the new one is proven.

FAQ

Does ALTER ROLE ... PASSWORD terminate existing sessions?

The documentation describes the statement as changing the role’s attributes; it does not state that existing authenticated sessions are terminated. New connections must present the new password. Plan for existing sessions to continue until they disconnect naturally.

Do I need to restart PostgreSQL after changing primary_conninfo?

The documentation states that if primary_conninfo is changed while the WAL receiver is running, that process is signaled to shut down and expected to restart with the new setting (PostgreSQL 18, Replication). A reload is the documented path; a full server restart is not required for this parameter.

What is the default for max_slot_wal_keep_size?

-1, meaning replication slots may retain an unlimited amount of WAL files (PostgreSQL 18, Replication). If you have not set a bound, a stalled slot has no ceiling.

Can a password rotation itself orphan a slot?

The retrieved documentation describes slot orphaning in terms of abrupt disconnection, failover, and slot synchronization — not password rotation as a direct cause. The practical risk is indirect: a credential mismatch prevents the standby from reconnecting, and the slot stops advancing. Treat the slot check as the control, not the password statement.

Does pgBackRest need reconfiguration during a password rotation?

The retrieved pgBackRest material does not establish that. pgBackRest’s archive-push is invoked by PostgreSQL’s archive_command, and its authentication is a separate concern from the replication role’s password. Verify your own configuration rather than assuming a change is required.

The native garment preview shows very small, centered lowercase meh. lettering on a dark shirt. No animal illustration is present. That is the whole design, and the restraint is the argument. The word sits near the center of the chest at a scale that asks you to lean in, and the period closes it with a flat stop rather than an exclamation or a wink. If you are looking for a meh shirt that commits to understatement rather than a cartoon mascot, this is the object to read closely.

We keep a small decision log for our own runbooks, and one recurring entry is the difference between a slogan and a usable promise. A slogan is loud, portable, and easy to repeat. A usable promise is specific enough that someone can check it. The meh. shirt is not a promise about uptime or incident response. It is a visual joke about lowered expectations, and its typography does the work: lowercase, tiny, punctuated, centered. The period is not decorative. It signals that the sentence is finished and that no further enthusiasm is coming.

What the artwork actually commits to

We noticed the scale first. Very small type on a dark shirt reads differently at conversation distance than it does in a mirror. Up close, the word is legible; across a room, it becomes a texture. That is a deliberate tradeoff. The design gives up billboard legibility in exchange for a private, almost deadpan register. It does not try to be a team jersey or a conference shirt. It is closer to a margin note.

The absence of an animal illustration matters too. A lot of low-effort humor shirts pair a word with a mascot, because the mascot carries the joke for anyone who cannot read the text at a distance. This design refuses that crutch. There is no sloth, no cat, no cartoon face to soften the message. The word has to stand alone, and the period has to carry the tone. That is a harder design to pull off, and it is why the shirt reads as minimalist rather than busy.

We should be precise about what the observation supports. The preview shows small lowercase lettering and no animal. It does not tell us how the shirt fits, how the fabric feels, or how the print holds up after washing. We have not worn or tested it, and we are not going to invent a wearing experience to fill the gap. The claim here is visual and editorial: the design’s meaning comes from scale, case, and punctuation, not from a mascot or a loud slogan.

Why the period does more work than the word

In our incident writeups, we try to end sentences with a period rather than an ellipsis. An ellipsis leaves the reader waiting for a resolution that may never come. A period says the thought is closed. The meh. shirt makes the same move. The word alone could read as a question or a shrug. The period makes it a verdict, delivered without heat. That is a useful tonal distinction for anyone who has ever written a postmortem that needed to be honest without being dramatic.

There is a work-culture reading here, but we want to keep it modest. Technical teams often oscillate between overclaiming and underclaiming. A status page that says everything is fine when it is not is an overclaim. A runbook that says ‘probably works’ is an underclaim. The meh. shirt sits in a third register: a deliberate, bounded expression of low enthusiasm. It is not a diagnosis of a system, and it is not a substitute for a real status update. It is a piece of clothing that says one small thing clearly and then stops.

The difference between a slogan and a usable promise

We keep coming back to this distinction because it applies to shirts as well as to runbooks. A slogan is designed to travel. A usable promise is designed to be checked. The meh. shirt is a slogan in the best sense: it is short, memorable, and visually coherent. It does not pretend to be a usable promise about reliability, security, or team performance. That honesty is part of its appeal. It is not asking to be taken as a statement about your engineering culture. It is asking to be read as a small, finished thought.

If you are selecting an off-duty object for a technical reader, the question is not whether the shirt demonstrates a software result. It does not, and it should not be asked to. The question is whether the design’s visual logic holds together. Here it does: the small lowercase type, the centered placement, the dark ground, and the period all point in the same direction. The shirt is not trying to be more than it is, and that restraint is the thing worth noticing.

What we’d do differently

If we were writing the product description, we would lead with the scale and the period rather than the word ‘funny.’ The humor is in the understatement, not in a punchline. We would also avoid implying that the shirt says anything about introversion as a personality type, because the artwork itself does not make that claim. The design is a small typographic gesture, and the most useful reading is the one that stays close to what is actually printed: lowercase, centered, punctuated, and finished.

Editorial note: This publication and Quirky Cotton Collective share an owner.

A DNS provider outage drill that changes nameservers is not a rehearsal. It is the incident, performed on a Tuesday because the calendar said so. For a lean team that is its own on-call rotation, the useful rehearsal is narrower and safer: exercise what you can observe, decide, and communicate while your authoritative provider is unreachable, and leave the delegation records alone.

This article describes that drill. It is a read-only exercise plus a tabletop. It does not validate failover, and it should not be reported as if it did.

Scope the drill before you schedule it

Two questions belong in a DNS provider outage drill, and neither requires touching production state:

1. Observation. When the provider’s control plane is unavailable, what do our checks actually show? Which queries fail, which are answered from cache, and which look healthy while being stale?
2. Decision. Who declares the outage, what evidence do they require, what will they refuse to do under pressure, and what do they escalate?

Changing NS records at the registrar is a separate activity with its own risk profile. AWS’s migration guide is explicit about the cost of getting it wrong: if you do not lower the NS record TTL, a domain could be unavailable on the internet for up to two days if something goes wrong during a migration (AWS, Making Route 53 the DNS service for a domain that’s in use). That is not a step to rehearse casually on a live zone. The drill described here deliberately excludes it.

Understand what you are simulating

Route 53’s documentation separates a control plane, which handles management operations such as creating and updating resources, from a data plane that provides core functionality (AWS, Resilience in Amazon Route 53). AWS recommends data-plane mechanisms such as health checks and ARC routing control for failover during disaster recovery, rather than relying on control-plane operations (AWS, Best practices for Amazon Route 53 DNS).

That distinction gives the drill its scenario. The realistic provider outage for most lean teams is not “every authoritative server stops answering.” It is “the console and API are unavailable or degraded, and we cannot make changes for a while.” Existing records keep being served from the provider’s data plane and from resolver caches. Your team is blind to the control plane and must decide whether to wait, escalate, or invoke a break-glass path.

Write the scenario down in those terms. Label it hypothetical. A scenario is not an incident report, and the drill artifact should say so.

Build the observation kit first

The drill is only as good as the evidence the on-call engineer can gather in the first ten minutes. Prepare the commands before the window, and have the on-call run them against live production records in read-only mode.

Compare authoritative and recursive answers

Query the provider’s authoritative nameservers directly, then query public resolvers, then query your own resolver. A practical troubleshooting guide recommends exactly this split: query multiple resolvers and authoritative servers directly rather than testing with only one resolver, and use dig +trace or direct queries against the authoritative nameservers to distinguish authoritative from recursive answers (Azion, DNS troubleshooting).

In a provider control-plane outage, the authoritative answers should still be correct. If they are, and a resolver returns old data, the guide’s rule applies: the issue is caching or resolver-related, not authoritative (Azion). That distinction is the single most useful thing the on-call can establish during the drill, because it determines whether the team waits or escalates.

Capture TTL and negative caching behavior

Resolvers cache responses for the TTL, and some caches also retain negative answers for a period (Azion). During the drill, record the TTLs on the records that matter — apex A/AAAA, the NS set, MX, and any health-checked failover records — and note which resolvers return which remaining TTL. This is observation, not measurement of failover time.

RFC 1034 explains why stale answers are expected rather than anomalous: access to information is more critical than instantaneous updates or guarantees of consistency, and when updates are unavailable due to network or host failure, the usual course is to believe old information while continuing efforts to update it (RFC 1034). A drill that treats cached answers as a bug will produce the wrong decision.

Trace the delegation chain

Run dig +trace for one or two critical names and record where the chain resolves from cache and where it reaches the provider. This exercises the team’s ability to answer the question “is the delegation intact?” without changing it. It also surfaces the parent-zone TTL issue that AWS documents: some full resolvers might cache the TTL of the NS record of the parent authoritative server, so that TTL must also be reduced during a real migration (AWS). The drill does not change that TTL; it records what it currently is and who owns it.

Timestamp everything

Every command in the kit should be run with a timestamp in the drill log. The timestamps are the raw material for the decision record, and they are the only honest basis for any later claim about how long the team took to reach a conclusion.

Run the drill as a tabletop plus read-only probing

Pick a 60–90 minute window. Announce it to the team as a drill, not an incident. The on-call engineer walks the existing runbook against live read-only queries while a second person plays the provider’s status page and the escalation contact.

A workable sequence:

1. Inject the scenario. Read the hypothetical aloud: “The provider console and API have been unreachable for 20 minutes. Status page says investigating. No ETA.”
2. Observe. The on-call runs the observation kit and writes down what each check shows, with timestamps.
3. Decide. The on-call states, out loud, what they would do next and what evidence they used. The second person records the decision and the evidence.
4. Escalate. The on-call states who they would contact, in what order, and what they would say. No actual pages are sent unless the team has agreed to a separate escalation test.
5. Stop. At the end of the window, the drill ends. No production changes were made, and the log says so.

The runbook itself should already exist. If it does not, write the recovery checklist before the drill, not during it; the Recovery Checklist article covers that preparation step.

Measure the declaration gap in miniature

The most valuable artifact from this drill is a small, honest record of the gap between “we would declare this handled” and “we would have evidence that traffic moved.” A practitioner account of a real failover describes the pattern: the DNS record changed in seconds, but traffic moved 18 minutes later, and the author argues that failover is complete only when traffic distribution changes, not when the record changes (Rack2Cloud, DNS failover testing). The same account argues that a test confirming the record updated and the health check passed has validated the protection plane, not whether traffic actually moved (Rack2Cloud).

That account is one anecdote, not a benchmark. Do not import its 18 minutes into your own RTO model. Instead, use the distinction it draws as a checklist item: during the drill, ask the on-call to state separately (a) when they would declare the provider issue resolved and (b) what evidence they would require before believing traffic had returned to normal. Record both answers and the gap between the evidence they named and the evidence they actually have access to.

If the team cannot measure traffic distribution at all, the drill should record that as an open question and schedule the measurement separately. A read-only drill does not validate failover capability. It validates observation and decision. Saying otherwise in the postmortem is how a team convinces itself it has tested something it has not.

Write the decision record

The drill’s output is a short decision record, not a narrative. It should answer, in plain sentences:

• Who declares a DNS provider outage, and what evidence do they require before declaring it?
• What will the team not do under pressure — for example, change NS records at the registrar without a second approver, or edit records repeatedly during the incident? The troubleshooting guide’s advice is to stabilize, lower TTL before planned migrations, and change once, rather than changing records repeatedly during an incident (Azion).
• What is the escalation path, and who owns the provider relationship?
• Which signals were missing during the drill, and which of them can be added before the next one?
• What remains hypothetical, and what would need to be measured to turn it into a fact?

File the record where the team keeps runbooks and decision records, and link it from the runbook. The next drill should start by reading it.

Keep the drill honest

Three rules keep a no-change DNS drill from turning into folklore:

Label hypotheticals as hypotheticals. The scenario is invented. The commands were real. The timestamps are real. The conclusions about what the team would do are decisions, not observations. Write them that way.

Do not claim measured RTOs you did not measure. A drill that never moved traffic cannot report a recovery time. It can report how long the team took to reach a decision, which is a different and still useful number.

Do not present the drill as a failover test. The distinction between record change and traffic movement is the point. A drill that blurs it teaches the team the wrong lesson at the worst possible moment.

What this drill does not cover

It does not exercise registrar access, DNSSEC key changes, or secondary-provider cutover. Those involve production state changes and need their own preparation, approvals, and rollback plans. AWS’s migration guide is the right starting point for the cutover itself, including its warning that DNSSEC signing cannot be enabled across two providers at the same time and that the DS record must be removed from the parent zone before migration (AWS). None of that belongs in a read-only rehearsal.

What the drill does cover is the part most lean teams skip: the ten minutes before anyone touches anything, when the on-call has to decide whether the problem is authoritative, recursive, or somewhere in between, and whether to wait or escalate. That decision is worth rehearsing. The nameservers can stay where they are.

Questions teams ask before running this

Can we run this during business hours?

Yes. The drill makes no production changes. The main constraint is that the on-call should not be simultaneously handling a real incident, so pick a window when the rotation is quiet and announce it in advance.

Do we need a second DNS provider to run it?

No. The drill is about observation and decision during a control-plane outage. A secondary provider is a separate architectural decision with its own rehearsal requirements.

How do we know the drill was useful?

If the decision record names at least one missing signal, one unclear owner, or one runbook step that could not be followed as written, the drill did its job. A drill that produces no gaps usually means the team did not probe hard enough.

Should we lower TTLs before the drill?

No. Lowering TTLs is a production change and belongs to a planned migration, not a read-only rehearsal. AWS recommends temporarily shortening TTLs when making changes to critical DNS entries so changes can be observed and rolled back quickly (AWS), but that advice applies to the change window, not the drill.

Editorial note: Gray Haven Lab and Quirky Cotton Collective share an owner.

The garment image shows a black-and-white speckled rectangle, a dark binding strip down the left edge, and a white ruled label headed COMPOSITION. Inside that label, an outlined ghost and pumpkin sit on the ruled lines. The whole thing is printed on a shirt. It is not a physical notebook sewn to fabric, and it is not a slogan with a notebook drawn beside it. The design commits to being a cover.

That distinction matters more than it first appears. Most group costumes for teachers rely on a phrase — Teacher Squad, Boo Crew, something in a friendly script — and the shirt becomes a billboard for the joke. This one makes the shirt a surface. The speckling, the binding, the ruled label: those are the visual grammar of a composition book, and the ghost and pumpkin are drawn as if they belong on that label, not pasted over it. The costume works because the object is recognizable before the joke lands.

We keep coming back to the label. A composition notebook label is a small piece of real estate with a specific job: identify the contents. In a classroom, that label is where a student writes a name, a subject, a period. It is a boundary marker. The printed version here replaces the handwritten name with a ghost and a pumpkin, which is a quiet substitution — the label still does its job of saying this is the thing, but the thing is now a costume. The ruled lines behind the ghost are not decoration; they are the lines that would hold handwriting. The design trusts the viewer to know what those lines are for.

That is a different kind of visual argument than a slogan makes. A slogan asserts. A surface invites. When a teacher wears a shirt that reads as a notebook cover, the joke is not I am a teacher — it is I am the object you write in. The wearer becomes the thing that gets filled up, annotated, and carried around. For a group costume, that is a more flexible premise than a matching phrase, because the notebook cover is a familiar shape across grade levels and subjects. It does not require everyone to agree on a caption.

We noticed the ghost and pumpkin are outlined rather than filled. That choice keeps the label legible. A solid black ghost would swallow the ruled lines and turn the label into a silhouette. The outline lets the lines show through, so the label still reads as a label. It is a small piece of design discipline: the costume elements sit inside the existing structure instead of replacing it. The same is true of the speckling — it is black-and-white, not colored, so the notebook cover stays recognizable as a composition book rather than drifting into generic Halloween pattern.

The shirt is listed as Composition Notebook Costume Teacher Group Halloween by Quirky Cotton Collective. The retail title uses the word costume, and that is accurate in the sense that the printed artwork is doing the work of a costume piece. It is not a physical notebook, and the design does not pretend otherwise. The ghost and pumpkin are printed on the label area, which means the costume reads at conversation distance without requiring accessories. That is a practical advantage for a group outfit: the shirt carries the visual weight, and the rest of the outfit can stay simple.

What we find useful about this design as an object is the way it separates surface from message. A lot of group shirts collapse the two: the message is the whole point, and the surface is whatever fabric the message is printed on. Here, the surface is the point. The composition notebook cover is a writing surface, and the ghost and pumpkin are the only writing on it. The design does not tell you what to think about teachers or Halloween. It gives you a familiar object with a small seasonal mark, and lets the recognition do the rest.

That is also where the design stops. It does not claim to improve anything, and it does not need to. The artwork is a printed notebook cover with a label, a ghost, and a pumpkin. The binding strip is dark, the speckling is black-and-white, the label is white with ruled lines. Those are the observed elements. The interpretation — that the shirt works as a costume because it treats the body as a cover rather than a billboard — is ours, not a historical claim about composition notebooks or a statement about what the shirt does for the wearer.

For a group costume, the practical question is whether the design holds up when several people wear it at once. A surface-based design tends to hold up better than a slogan-based one, because the eye reads the shape before it reads the words. The composition notebook cover is a shape. The label is a shape. The ghost and pumpkin are small marks inside that shape. When five people wear the same shirt, the repetition reads as a set of identical objects, which is the visual logic of a group costume. A slogan would read as five people saying the same thing. The notebook cover reads as five notebooks.

We would not overstate this. The shirt is a printed garment, and the artwork is what the image shows. It is not a notebook, and it does not turn the wearer into one. But the design makes a clear choice about where the joke lives: in the surface, not in the caption. That choice is what makes the costume legible across a group, and it is why the label — with its ruled lines and its small seasonal marks — does more work than a slogan would.

What we’d do differently

If we were laying out a group costume around this shirt, we would treat the label as the anchor and keep everything else quiet. The design already commits to a surface; adding more printed text or extra props would compete with the ruled lines and the outlined ghost. We would also resist the urge to explain the joke on the shirt itself. The composition notebook cover is recognizable enough that the costume does not need a caption, and the label is stronger when it is allowed to be a label rather than a place for a punchline.

The native garment preview shows a dark shirt with large, bold, white lowercase lettering: meh. There is no animal illustration, no secondary graphic, no decorative frame. Just four characters and a period, set at a size that would be legible across a room. The word is dismissive. The typography is not. That mismatch is the entire design, and it is worth reading closely because it says something specific about how terse messages behave in technical teams.

We keep a small decision log for our own operations work, and one recurring entry is the difference between a slogan and a usable promise. A slogan is a phrase a team puts on a wall. A usable promise is a sentence with a trigger, an owner, and a next step. The meh. shirt is not a promise. It is the opposite: a refusal to promise anything, printed at a volume that makes the refusal impossible to miss. That is a legitimate piece of communication, and it is worth understanding why it works before deciding whether it belongs in your rotation.

What the artwork actually commits to

Look at the period first. Without it, meh is a sound, a shrug, an open-ended noise. With it, the word becomes a complete sentence — a verdict, delivered and closed. The period is the smallest element on the shirt and the one doing the most grammatical work. It converts ambivalence into a statement. That is a subtle move, and it is the reason the design reads as dry rather than merely lazy.

The lowercase matters too. Capital letters would make MEH. an announcement, almost aggressive. Lowercase keeps it conversational, a muttered aside rather than a declaration. The bold weight and the large scale pull in the opposite direction, giving the word a visual confidence the sentiment does not have. The shirt is loud about being unimpressed. That tension is the joke, and it holds up because the design does not try to explain it.

We would describe the palette plainly: white lettering on a dark ground, high contrast, no gradients or outlines. The retail title mentions sarcasm, introversion, and dry humor, but those are marketing categories, not printed elements. What is actually printed is the word and the period. Everything else is interpretation, and we should be clear about which is which.

Why terse messages are hard to get right on a team

In a small on-call rotation, the messages that cause the most trouble are rarely the long ones. They are the short ones that leave the reader guessing about intent. A one-word reply in a incident channel can read as calm, dismissive, or hostile depending on who sent it and what came before. The meh. shirt is interesting precisely because it makes that ambiguity explicit and then owns it. It is not a failed attempt at clear communication. It is a deliberate choice to communicate very little, at high volume.

That is a useful thing to have a vocabulary for. When we review a postmortem and find a timeline entry that says only “acknowledged” or “looking,” we ask the same question the shirt raises: is this terse because the situation is genuinely simple, or because the sender has not yet decided what they think? The shirt does not answer that question. It just makes the posture visible.

The object itself

The shirt is listed as the Meh Funny Sarcastic Saying Introvert Dry Humor tee from Quirky Cotton Collective. The design is a single typographic statement, which means it will read the same whether you pair it with work clothes or weekend clothes. There is no illustration to date it, no reference to a specific platform or meme cycle, and no secondary text that would tie it to a particular moment. That is a point in its favor for anyone who wants an off-duty object that does not require explanation.

We are not going to claim we wore it, tested it, or observed reactions to it. What we can say is that the design is legible at the scale shown in the preview, and that the single-word-plus-period construction is the kind of thing that either lands immediately or does not land at all. There is no middle ground with a shirt like this, which is arguably the point.

Where the design sits in a work-culture conversation

Technical teams spend a lot of energy on wording. Runbook steps, escalation messages, status updates, and decision records all depend on saying exactly enough and no more. The meh. shirt is a useful counterexample because it says exactly enough to be understood and no more than that. It is not a model for incident communication. It is a reminder that brevity is a tool, and like any tool it can be used well or badly.

We would not put this shirt in the same category as a diagram joke or a boundary-setting phrase. Those designs usually carry a small argument about how to work. This one carries a small argument about how to decline to argue. That is a different move, and it is worth naming as such rather than lumping it into a generic sarcasm category.

What we’d do differently

If we were writing the retail copy, we would drop the list of personality labels and describe the design instead: large lowercase meh. with a period, white on dark, no illustration. The labels invite the reader to project a persona onto the shirt, when the artwork is doing something more specific and more interesting. The period is the story. The rest is packaging.

Editorial note: This publication and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The shirt reads I HAD POTENTIAL in white capitals, and beneath the words sits a small physics diagram: a curved ramp descending from an upper-left ledge to a lower-right flat ledge. At the top of the ramp, on the upper ledge, there is a dashed empty circle. At the bottom, resting on the flat, there is a solid white ball. The ramp is curved, not a straight diagonal. That curvature matters more than the joke first suggests.

Most readings of this shirt stop at the pun. Potential is the word we use for what someone might have become, and it is also the word physics uses for stored energy. The dashed circle at the top is where the ball would have been if it had been placed there. The solid ball at the bottom is where it actually is. The joke is that the ball had gravitational potential energy and then spent it, and the shirt is wearing the aftermath.

But the drawing is more specific than the pun. A dashed circle is not a ball that fell. It is a position that was never occupied, or a position that is now empty. The solid ball at the bottom is not drawn mid-roll or mid-bounce. It is at rest on the lower ledge. The diagram shows a completed transfer, not a failure in progress. That is a different claim from “I had potential and wasted it.” It is closer to “the potential was real, and it has already been converted into something else.”

What the curved ramp actually shows

In a simple physics diagram, a ball at the top of a ramp has gravitational potential energy relative to the bottom. As it rolls down, that potential energy becomes kinetic energy, and friction turns some of it into heat. By the time the ball rests on the lower ledge, the energy has not vanished. It has been redistributed. The shirt does not draw the heat, the sound, or the deformation of the ball. It draws only the before-position and the after-position, with the ramp connecting them.

That is a useful visual argument about work and careers, and it is not the argument most sarcastic shirts make. The usual version of this joke is a before-and-after of ambition: a promising person on one side, a tired person on the other. This shirt skips the person entirely. It gives you a dashed outline and a solid object. The dashed outline is the shape of an expectation. The solid object is the thing that actually exists.

We read the dashed circle as the place where the ball was supposed to be, not as a ghost of the ball itself. The distinction is small but it changes the tone. A ghost implies loss. An empty position implies a system that had a slot for something and did not fill it. The shirt is less “I am a disappointment” and more “the diagram has a missing element.” That is a more interesting object to wear, because it invites the reader to ask what filled the slot instead.

The difference between a slogan and a usable promise

Technical teams spend a lot of time around the word potential. It shows up in performance reviews, in architecture discussions, and in the quiet gap between what a system was designed to do and what it currently does. A service has potential capacity. A backup has potential recoverability. A runbook has potential usefulness. The word is almost always a claim about a state that has not been tested.

The shirt is funny because it treats potential as a quantity that can be spent. That is closer to how operations actually works than the motivational version. A system does not have potential in the abstract; it has a specific amount of headroom, a specific recovery time, a specific margin before the next failure. When we say a service “has potential,” we usually mean we have not measured it. The dashed circle is the unmeasured state. The solid ball is the measurement.

This is why the shirt works as an off-duty object for people who carry a pager. It is not a work shirt. It does not name a tool or a failure mode. But it does make a small, precise joke about the difference between an intended state and an observed state, and that is the same distinction behind every useful monitoring check. The diagram is not evidence of anything. It is a drawing. But it is a drawing that gets the physics-adjacent vocabulary right, and that is more than most slogan shirts manage.

Reading the object without overclaiming it

The shirt is sold by Quirky Cotton Collective under the title “I Had Potential Funny Physics Pun Sarcastic Minimalist Quote.” The retail description is not the artwork. The artwork is the white text and the white ramp-and-ball diagram on the garment. We are describing what is visible in the retail image, not what the shirt proves about the wearer, the brand, or the state of the world.

If you want the object itself, it is listed here: I Had Potential Funny Physics Pun Sarcastic Minimalist Quote. The listing is the source for the design observation, and the design observation is the only thing we are treating as established. We have not worn it, tested it, or surveyed anyone who has. The argument here is about the drawing, not the fabric.

The drawing is worth the attention because it commits to a specific visual grammar. The dashed circle is not decorative. The solid ball is not a punchline. The curved ramp is not a straight line, and that curve is what makes the diagram read as a physical path rather than an arrow from “then” to “now.” A straight diagonal would have suggested a trend. A curve suggests a trajectory with a shape, which is a more honest picture of how most potential gets spent.

What we’d do differently

If we were designing a shirt about operational potential, we would probably make the same mistake the slogan version makes: we would put the words first and the diagram second. This shirt does the opposite. The words are large, but the diagram is the part that rewards a second look. The lesson for our own runbooks and dashboards is that the diagram is usually the argument, and the label is usually the summary. We would spend more time on the shape of the drawing and less on the wording above it. The dashed circle is doing more work than the capital letters.

A backup job that exits 0 has told you one thing: the process finished. It has not told you that the snapshot contains the files you meant to protect, that the bytes are intact, or that the result can be restored. For a team of two to fifteen engineers who are also the on-call rotation, those are three different questions, and the exit code answers none of them.

The failure mode worth designing against is quiet. A backup pointed at a mount that came up empty, a path that changed after a deploy, an exclude pattern that grew broader than intended, a snapshot that captured only the directory skeleton — all of these can produce a successful run, a valid repository, and a clean checksum. The job reports success because the process did what it was told. The payload is hollow.

The cheapest defense is two assertions that fail for different reasons, scheduled on different cadences: a content assertion that proves the expected files are present, and an integrity assertion that proves the bytes are intact. A canary file makes the content assertion trivial to automate.

Layer 1: Assert what is actually in the backup

Content assertion means inspecting the latest backup and checking for paths you expect. It is not the same as checking that the job ran. It is a query against the artifact.

With restic, the ls command lists the files in a specific snapshot, and the special snapshot ID latest selects the most recent one. The command supports a --json flag for scripting, which is what you want if the check is going to run unattended and fail a pipeline. A minimal assertion looks like this:

restic -r "$REPO" ls latest --json \
  | jq -e '.[] | select(.path == "/srv/app/canary.txt")' \
  > /dev/null

If the path is absent, jq -e exits nonzero and the job fails. That is the whole mechanism. The rest is deciding what to assert on.

For PostgreSQL base backups taken with pg_basebackup, the equivalent primitive is the server-generated backup_manifest. pg_verifybackup reads that manifest and checks that the data files on disk match what the server intended to send, including a stage that checksums every file and compares against the manifest values. That is a content-and-integrity check in one tool, and it is worth running against every base backup before you trust it. The PostgreSQL documentation is explicit that this validation does not and cannot include every check a running server performs, and that test restores remain necessary. Treat pg_verifybackup as a floor, not a ceiling.

One caveat on restic specifically: excludes do not apply to backup sources explicitly passed on the command line. If your job runs restic backup /srv/app, an exclude for /srv/app/canary.txt will not remove it. That is a useful property for canary design — the canary survives even a broad exclude — but it also means an exclude-based scope error will not be caught by the canary alone. Assert on a second expected path that is subject to your exclude rules.

Layer 2: Assert the bytes are intact

Integrity assertion answers a different question: are the stored bytes the bytes that were written? This is where checksums and read-back verification live.

restic’s check command tests the repository for errors. By default it verifies repository structure without reading all data. Adding --read-data makes it download and verify the actual backed-up data, not just the index. That is the difference between “the catalog is consistent” and “the books are readable.” For repositories large enough that a full read is impractical on a schedule, --read-data-subset lets you verify a fraction per run and spread coverage across a rotation. The restic community discusses this tradeoff openly; the pattern of checking a subset per night so that the full repository is covered over a week or a month is a practitioner convention, not a documented guarantee, and it should be treated as such.

For PostgreSQL, pg_verifybackup performs the checksum stage against the manifest. Note that --skip-checksums exists and is faster precisely because it does not read the files — it checks presence and size only. If your verification job uses that flag to save time, you have moved from integrity assertion back to content assertion and should know it.

Object stores give you a transport-level integrity signal, which is useful but narrower than it looks. Amazon S3 accepts a checksum algorithm on upload, calculates its own server-side value, and validates the client-provided value before storing the object. Google Cloud Storage supports CRC32C and MD5, recommends CRC32C, and rejects a write with a 400 error when the client-supplied checksum does not match the server-calculated one. These mechanisms catch corruption in transit and at write time. They do not tell you that the object contains the data you expected, and they do not tell you the backup is restorable. A canary file plus a content assertion covers the first gap; a restore test covers the second.

The canary file

A canary is a small file with known content, written into each backup scope before the backup runs, then asserted present and byte-identical in the latest backup. It is not a feature of restic or pgBackRest. It is a practice built on the primitives above.

The design is deliberately boring. Before the backup job starts, write a file containing a timestamp, a hostname, and a fixed string to a known path inside the backup scope. After the backup completes, list the latest snapshot and confirm the path exists. On the restore side, extract the file and compare its checksum to the value recorded at write time.

# before backup
printf '%s %s %s\n' "$(date -u +%FT%TZ)" "$(hostname)" "grayhaven-canary-v1" \
  > /srv/app/canary.txt
sha256sum /srv/app/canary.txt > /srv/app/canary.sha256

# backup
restic -r "$REPO" backup /srv/app

# content assertion
restic -r "$REPO" ls latest --json \
  | jq -e '.[] | select(.path == "/srv/app/canary.txt")' > /dev/null

What this catches: an empty mount, a wrong path after a deploy, a backup that captured only directory metadata, a snapshot that silently excluded the application directory. What it does not catch: a backup that contains the canary but is missing other expected data. That is why the canary is a floor, not a complete content assertion. Pair it with at least one assertion on a real application path.

There is a subtlety with restic’s change detection. restic uses file metadata — mtime, ctime, size, inode on Unix — to decide whether a file is likely unchanged, and skips rescanning it. If the canary is rewritten every run with a new timestamp, its metadata changes and it will be rescanned. That is what you want. If you write the canary once and never touch it, restic may treat it as unchanged and the canary stops proving anything about the current run. Rewrite it every time, or use --force on a periodic basis to turn off change detection entirely.

Wiring it into the rotation

The scheduling split matters more than the tooling. Content assertions are fast and cheap; run them on every backup and let a failure page. Integrity assertions are slow and expensive; run them on a subsetted cadence in the background and let a failure open a ticket. A team of five engineers cannot afford a nightly full read-back of a multi-terabyte repository, and it does not need one. It needs the cheap check to be reliable and the expensive check to be regular.

Concretely: the backup job writes the canary, runs the backup, lists the latest snapshot, asserts the canary path and one application path, and exits nonzero on any failure. That exit code is what your monitoring watches. The integrity pass runs separately, on a schedule that covers the repository over time, and reports into a queue rather than a pager. If the integrity pass finds corruption, you have time to act; if the content assertion fails, you are already backing up nothing and need to know now.

For PostgreSQL, the same split applies. Run pg_verifybackup against each base backup as part of the backup job — it is the content-and-checksum check — and keep the full restore rehearsal on a slower cadence. The PostgreSQL documentation is direct that verification is not a substitute for test restores.

What this does not prove

A passing canary and a clean checksum rule out emptiness and corruption. They do not rule out a backup that cannot be brought up: a missing WAL segment, a version mismatch, a restore procedure that references a path that no longer exists. Those failures live in the restore path, and the only way to find them is to restore.

That is why the restore lottery stays a separate ritual. Pick a backup at random, restore it to a scratch environment, and assert that the application starts and the canary is present with the expected checksum. The canary does double duty here: it is the content assertion during backup and the end-to-end assertion during restore. If the restored canary matches the checksum recorded at write time, you have evidence that the bytes survived the full round trip.

If you have not written down the restore steps yet, do that before adding more checks. A verification job that nobody knows how to act on is its own kind of hollow backup. The recovery checklist is the companion artifact to this one: the checks tell you whether to reach for it, and the checklist tells you what to do when they fail.

A checklist for this week

• Write a canary file into each backup scope before the job runs, with a timestamp and a fixed string. Rewrite it every run.
• Add a content assertion to the backup job: list the latest snapshot and fail if the canary path or one application path is missing.
• Confirm the assertion actually fails when it should. Point the job at an empty directory once and watch it exit nonzero.
• Schedule an integrity pass — restic check --read-data-subset or pg_verifybackup — on a cadence you can sustain, and route its failures to a ticket, not a page.
• Keep the restore lottery separate. Restore a random backup, start the application, and compare the restored canary checksum to the recorded value.

The exit code is a statement about the process. The canary and the checksum are statements about the payload. Keep them separate, schedule them differently, and let each fail for its own reason.

Four root vegetables sit in a row, smiling. A carrot, a radish, a beet, an onion. Their roots are drawn down into the soil, visible, not hidden. Around them, in a ring, the words: LET’S ROOT FOR EACH OTHER AND WATCH EACH OTHER GROW.

The pun is immediate. Root as in the thing the vegetables are doing underground, and root as in the thing a crowd does for a team. The shirt wants both at once, and for a moment it gets them. But the longer I look at the artwork, the more I think the two meanings are not actually doing the same work, and the design quietly picks a side.

This is a column about reading the object, not a review of how it wears. The listing is Let’s Root For Each Other And Watch Each Other Grow Garden from Quirky Cotton Collective. The visual observation is the artwork itself: the four smiling vegetables, roots in soil, the sentence wrapping around them. Everything below is interpretation of that graphic, not a claim about the shirt’s history, popularity, or effect on anyone’s team.

What the artwork shows, literally

The sentence is not a caption underneath the vegetables. It surrounds them. That placement matters. A caption says here is a picture, and here is what it means. A ring says the meaning encloses the thing. The vegetables are inside the promise, not beneath it.

The roots are drawn. That is the unusual choice. Most gardening graphics show the leafy top and let the soil be a brown line. Here the carrot’s taproot, the radish’s small white tail, the beet’s stain, the onion’s fibrous threads are all visible. The design is insisting on the underground half. The part you don’t see from above is the part being celebrated.

And the faces are smiling. Not grinning, not winking. Smiling the way a diagram smiles when someone has decided the diagram should be friendly. Four different vegetables, four different shapes, one shared expression. That is a visual argument about similarity inside difference, and it is the strongest thing the artwork does.

The two meanings of rooting do not agree

To root for someone is to want their success from the outside. It is spectatorship with goodwill. You are not in the soil with them. You are in the stands. The relationship is asymmetric: they perform, you applaud.

To root, as a plant does, is to be the one doing the work underground. It is not spectatorship. It is the slow, unglamorous, mostly invisible business of taking hold. Nobody claps for a taproot.

The shirt’s sentence uses both meanings in one line, and the pun works because the two are close enough to blur. But the artwork does not blur them. It draws the roots. It puts the vegetables inside the sentence. It makes the underground visible. The design is not saying we will cheer for you. It is saying we are all down here, doing the same unglamorous thing, and that is what we have in common.

That is a different promise than the one the words alone make. The words alone could be a slogan on a mug. The artwork makes it a claim about shared work.

Why the second half of the sentence is the harder half

AND WATCH EACH OTHER GROW. The first half is easy to say. Rooting for someone costs nothing. Watching them grow costs something, because growth is uneven, and watching it means noticing when someone else’s root goes deeper than yours.

The artwork handles this better than the text does. Four vegetables, four different root shapes. The carrot’s taproot is long and singular. The radish’s is short and round. The beet’s is bulbous and stained. The onion’s is a fringe. None of them is the correct root. The design does not rank them. It just shows them all, side by side, in the same soil, with the same smile.

That is the visual answer to the uncomfortable question the sentence raises. If we are watching each other grow, what happens when one of us grows faster? The artwork’s answer is: the soil holds all four shapes. The smile is the same. The roots are not.

The difference between a slogan and a usable promise

I keep a small decision log for the teams I work with. One of the entries I return to most is a note about the difference between a value and a mechanism. A value is a sentence everyone agrees with. A mechanism is what happens on the Tuesday when the value is inconvenient.

LET’S ROOT FOR EACH OTHER is a value. It is easy to agree with. The artwork, though, is closer to a mechanism, because it shows the thing that makes mutual support possible: visible roots. The underground work is drawn. It is not left to the imagination. If you can see the root, you can see who is doing the unglamorous part, and you can stop pretending that the visible leafy part is the whole plant.

That is a real observation about the design, not a claim that wearing the shirt will make a team function better. The shirt is a shirt. But the artwork contains an argument, and the argument is more specific than the sentence it wraps around.

What the ring of text does to the reader

There is a small formal trick in the layout. The sentence surrounds the vegetables, which means the reader’s eye has to travel around the group to finish the thought. You cannot read the whole line in one glance. You have to go around. That is a mild but real enactment of the sentence’s content: to root for each other, you have to keep the whole group in view, not just the one in front of you.

I would not overstate this. It is a ring of text on a shirt, not a diagram of a distributed system. But the choice to wrap rather than caption is the kind of decision that separates a design with a point of view from a design that just places words above a picture. The wrap makes the group the unit. The caption would have made the sentence the unit.

What we’d do differently

If I were reading this artwork as a model for how to phrase a team promise, I would take one thing from it and leave one thing behind. I would take the visible roots: the insistence that the unglamorous, underground half of the work is part of the picture, not a footnote. I would leave behind the smile, or at least I would ask what the smile is doing. Four identical smiles on four different vegetables is a claim that the emotional experience is the same for everyone. It probably is not. The carrot and the onion are not having the same day.

The shirt does not have to solve that. It is a shirt. But the artwork is more interesting than its sentence, and the interesting part is the part that shows the roots. That is the part worth keeping.

Editorial note: this publication and Quirky Cotton Collective share an owner.