Why a lottery beats a checklist

A restore checklist tends to test the backup you already know is easy. The snapshot you took last night, the one whose restore path you walked through during onboarding, the one on the host whose credentials are still in your shell history. That is not the backup an incident will hand you. An incident hands you whatever exists in the repository at the moment you need it, including the snapshot from a host you decommissioned two quarters ago and the one whose encryption key was rotated without a note.

A weekly restore lottery replaces the hand-picked backup with a random draw from the repository’s own listing. The randomness is the point: it removes the selection bias that makes drills feel reassuring while leaving the real recovery path untested. The hard timeout is the second half of the design. A restore that exits zero after ninety minutes against a thirty-minute budget is a failed drill, not a pass, because the question is not whether the tool works but whether your team can recover within a window it can actually tolerate.

This is a rehearsal of the recovery decision, not a test of the backup vendor. Treat it that way and the drill stays small enough to survive a busy week.

Define the two random inputs before the drill

Two things get decided before anyone touches a terminal: the snapshot and the timeout. Both are recorded in the drill log before the restore starts. Neither is adjusted mid-drill.

The snapshot is drawn from the repository listing, not from memory or a spreadsheet. For restic, restic snapshots lists every snapshot with its ID, timestamp, host, tags, and directory, and the listing can be filtered by --path, by --host, or by combining filters, and grouped with --group-by on host, paths, or tags. Draw from that output. If you keep a spreadsheet of “known good” snapshots, you are not running a lottery; you are running a checklist with extra steps.

For pgBackRest, the equivalent is pgbackrest info against the stanza. The restore command defaults to selecting the latest backup from the first repository where backups exist, and --set accepts latest or a specific backup name. The lottery should draw a specific set name, not latest, because latest is the backup you would have picked anyway.

The timeout is a wall-clock budget agreed in advance, derived from what the team can tolerate during an incident rather than from how long the restore usually takes. It does not change between weeks, because changing it destroys the comparison. If the budget turns out to be wrong, change it deliberately between drills and note the change in the log.

Example: drawing the snapshot

# restic: list snapshots, then pick one at random
restic -r s3:s3.amazonaws.com/example-restic snapshots --json \
  | jq -r '.[].short_id' \
  | shuf -n 1

Record the chosen ID and its timestamp in the drill log before starting the restore. The same pattern works for pgBackRest: parse pgbackrest info --output=json, extract the backup labels for the stanza, and draw one. The tool’s own listing is the source of truth for what exists; a stale inventory is not.

Set the timeout as a hard stop

Start a timer when the restore command starts. Stop the drill when the timer expires, regardless of how close the restore looks to finishing. A timeout is a finding, not a failure to hide. The drill log records it as one of three outcomes: completed within timeout, completed but over timeout, or timed out. The third outcome is the most informative, because it tells you the recovery path has a step that does not fit inside the window you would actually have during an incident.

Do not extend the timeout to see whether the restore would have finished. That converts a bounded operational fact into an unbounded curiosity, and it teaches the team that the budget is negotiable. If the restore is still running when the timer expires, stop it, record where it was, and clean up the target.

Run the restore into an isolated target

The drill restores into a target that is not production and not reachable from production. For restic, restic restore <snapshotID> --target /srv/restore-drill/<date> extracts the snapshot to a directory; the special snapshot ID latest exists but the lottery should not use it. For pgBackRest, the restore command writes into the PostgreSQL data and tablespace paths, which by default are expected to be present but empty; the --delta option performs a delta restore using checksums, and --target-action controls what the cluster does when a recovery target is reached, with pause as the default.

Isolation matters for two reasons. First, a restore that writes into a live cluster’s data directory is not a drill; it is an incident. Second, the isolated target is where you can afford to let the restore run to its timeout without worrying about what it is overwriting.

Verify restored content, not just exit status

An exit status of zero means the restore command completed, not that the restored data is correct. restic’s restore command has a --verify flag described as “verify restored files content”; per the restic maintainers, it enables an additional check after the restore has been done, which goes over all files, reads them again, and verifies that the correct data is read. It is mutually exclusive with --dry-run. It is not a full repository integrity check, and it does not concern itself with additional files in the target tree; it checks the files the snapshot lists.

For PostgreSQL, verification is a different shape. The restore is only meaningful if the cluster can start and replay WAL to a consistent state. PostgreSQL’s continuous archiving documentation is explicit that to recover successfully using continuous archiving, you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup. A restore that produces a data directory but cannot find the WAL segment it needs is a failed drill even if the restore command exited zero. The drill should include starting the restored cluster and confirming it reaches consistency, or at minimum confirming that the WAL archive contains the segments the backup’s history file names.

This is also where the drill catches archiving problems that monitoring missed. PostgreSQL’s documentation advises monitoring the archiving process to ensure it is working as intended; a restore lottery that regularly fails to find WAL is a stronger signal than a dashboard that says archiving is green.

Capture three outcomes and one change

The drill log is short by design. A long template invites skipping; a short one survives busy weeks. Each entry records:

• What was drawn: the snapshot ID, its timestamp, the host or stanza, and the repository.
• The budget: the wall-clock timeout agreed in advance.
• The outcome: completed within timeout, completed but over timeout, or timed out.
• Where time was lost, if any: credential retrieval, network path, target provisioning, WAL retrieval, verification.
• One concrete change for next week.

The “where time was lost” field is the one that earns its keep. A restore that times out during credential retrieval is a different problem from one that times out during WAL replay, and the fix for each is different. Recording the step, not just the total, lets next week’s drill target the slow part.

The “one concrete change” field keeps the drill from becoming a ritual. If the same step is slow three weeks running, the change should be to that step, not to the drill.

Keep the note blameless and the cadence honest

The drill log is a blameless artifact. It records what was tested, what the budget was, where it stalled, and what changes next. It does not record who ran the drill badly or whose credential was missing. The Google SRE Workbook’s chapter on postmortem culture makes the case that a truly blameless postmortem culture results in more reliable systems, and recommends starting small with a very basic procedure and then reflecting and tuning it. The restore lottery log is that basic procedure applied to recovery rehearsal: short, factual, and acted upon.

Two habits keep the cadence honest. First, rotate who runs the drill, so recovery knowledge is not concentrated in one person. The lottery is also a knowledge-distribution mechanism; the person who draws the snapshot and watches the timer learns where the recovery path actually goes. Second, do not skip the drill during busy weeks. The busy weeks are the ones the drill is for, because they are the weeks when the team’s recovery capacity is most likely to be degraded and the drill is the only signal that it has degraded.

If a week is genuinely too busy to run a full restore, run a smaller version: draw the snapshot, start the timer, and stop at the first step that requires a credential or a network path you do not have. That is still a finding, and it is still cheaper than discovering the gap during an incident.

What the lottery is not

The restore lottery is one rehearsal among several. It does not replace periodic full restore tests against production-shaped infrastructure, repository integrity checks, or monitoring of the backup and archiving pipelines. It does not prove the backup tool is correct; it proves the team can execute a recovery decision within a bounded window on a backup it did not choose. Those are different claims, and conflating them is how drills become theater.

It also does not require a dedicated SRE function. A two-person team can run it in under an hour if the target is pre-provisioned and the credentials are already in the drill runner’s hands. The hard timeout is what keeps it from expanding to fill the available time.

FAQ

Should the timeout be the same every week?

Yes, until you deliberately change it. The timeout is the comparison point across weeks; changing it mid-drill or week-to-week destroys the signal. If the budget turns out to be wrong, change it between drills and record the change.

What if the random snapshot is from a host we no longer run?

Restore it anyway. That is exactly the kind of backup an incident might hand you, and the drill is the cheapest place to discover that the restore path for a decommissioned host no longer works. If the snapshot is genuinely out of scope, note why in the log and draw again, but do not make a habit of redrawing.

Does restic’s --verify prove the backup is good?

No. It verifies that the restored files match the snapshot’s recorded content, after the restore has completed. It does not check the repository’s integrity as a whole, and it does not concern itself with files outside the snapshot. Pair it with a repository check if you want integrity assurance; the lottery is about recovery execution, not repository health.

How does this interact with pgBackRest’s --delta?

--delta performs a delta restore using checksums, which is useful when the target already contains a previous restore. For a drill, a clean target is simpler and produces a cleaner timing signal; use --delta when you are rehearsing the incremental path specifically, and note in the log that you did.

What if the restore finishes early?

Record the elapsed time and stop. Finishing early is a good outcome, not an invitation to add scope. The next week’s draw is the next test.

Related: Write the Recovery Checklist Before You Need It covers the pre-incident checklist that the lottery log feeds into.