A backup job that exits 0 has told you one thing: the process finished. It has not told you that the snapshot contains the files you meant to protect, that the bytes are intact, or that the result can be restored. For a team of two to fifteen engineers who are also the on-call rotation, those are three different questions, and the exit code answers none of them.

The failure mode worth designing against is quiet. A backup pointed at a mount that came up empty, a path that changed after a deploy, an exclude pattern that grew broader than intended, a snapshot that captured only the directory skeleton — all of these can produce a successful run, a valid repository, and a clean checksum. The job reports success because the process did what it was told. The payload is hollow.

The cheapest defense is two assertions that fail for different reasons, scheduled on different cadences: a content assertion that proves the expected files are present, and an integrity assertion that proves the bytes are intact. A canary file makes the content assertion trivial to automate.

Layer 1: Assert what is actually in the backup

Content assertion means inspecting the latest backup and checking for paths you expect. It is not the same as checking that the job ran. It is a query against the artifact.

With restic, the ls command lists the files in a specific snapshot, and the special snapshot ID latest selects the most recent one. The command supports a --json flag for scripting, which is what you want if the check is going to run unattended and fail a pipeline. A minimal assertion looks like this:

restic -r "$REPO" ls latest --json \
  | jq -e '.[] | select(.path == "/srv/app/canary.txt")' \
  > /dev/null

If the path is absent, jq -e exits nonzero and the job fails. That is the whole mechanism. The rest is deciding what to assert on.

For PostgreSQL base backups taken with pg_basebackup, the equivalent primitive is the server-generated backup_manifest. pg_verifybackup reads that manifest and checks that the data files on disk match what the server intended to send, including a stage that checksums every file and compares against the manifest values. That is a content-and-integrity check in one tool, and it is worth running against every base backup before you trust it. The PostgreSQL documentation is explicit that this validation does not and cannot include every check a running server performs, and that test restores remain necessary. Treat pg_verifybackup as a floor, not a ceiling.

One caveat on restic specifically: excludes do not apply to backup sources explicitly passed on the command line. If your job runs restic backup /srv/app, an exclude for /srv/app/canary.txt will not remove it. That is a useful property for canary design — the canary survives even a broad exclude — but it also means an exclude-based scope error will not be caught by the canary alone. Assert on a second expected path that is subject to your exclude rules.

Layer 2: Assert the bytes are intact

Integrity assertion answers a different question: are the stored bytes the bytes that were written? This is where checksums and read-back verification live.

restic’s check command tests the repository for errors. By default it verifies repository structure without reading all data. Adding --read-data makes it download and verify the actual backed-up data, not just the index. That is the difference between “the catalog is consistent” and “the books are readable.” For repositories large enough that a full read is impractical on a schedule, --read-data-subset lets you verify a fraction per run and spread coverage across a rotation. The restic community discusses this tradeoff openly; the pattern of checking a subset per night so that the full repository is covered over a week or a month is a practitioner convention, not a documented guarantee, and it should be treated as such.

For PostgreSQL, pg_verifybackup performs the checksum stage against the manifest. Note that --skip-checksums exists and is faster precisely because it does not read the files — it checks presence and size only. If your verification job uses that flag to save time, you have moved from integrity assertion back to content assertion and should know it.

Object stores give you a transport-level integrity signal, which is useful but narrower than it looks. Amazon S3 accepts a checksum algorithm on upload, calculates its own server-side value, and validates the client-provided value before storing the object. Google Cloud Storage supports CRC32C and MD5, recommends CRC32C, and rejects a write with a 400 error when the client-supplied checksum does not match the server-calculated one. These mechanisms catch corruption in transit and at write time. They do not tell you that the object contains the data you expected, and they do not tell you the backup is restorable. A canary file plus a content assertion covers the first gap; a restore test covers the second.

The canary file

A canary is a small file with known content, written into each backup scope before the backup runs, then asserted present and byte-identical in the latest backup. It is not a feature of restic or pgBackRest. It is a practice built on the primitives above.

The design is deliberately boring. Before the backup job starts, write a file containing a timestamp, a hostname, and a fixed string to a known path inside the backup scope. After the backup completes, list the latest snapshot and confirm the path exists. On the restore side, extract the file and compare its checksum to the value recorded at write time.

# before backup
printf '%s %s %s\n' "$(date -u +%FT%TZ)" "$(hostname)" "grayhaven-canary-v1" \
  > /srv/app/canary.txt
sha256sum /srv/app/canary.txt > /srv/app/canary.sha256

# backup
restic -r "$REPO" backup /srv/app

# content assertion
restic -r "$REPO" ls latest --json \
  | jq -e '.[] | select(.path == "/srv/app/canary.txt")' > /dev/null

What this catches: an empty mount, a wrong path after a deploy, a backup that captured only directory metadata, a snapshot that silently excluded the application directory. What it does not catch: a backup that contains the canary but is missing other expected data. That is why the canary is a floor, not a complete content assertion. Pair it with at least one assertion on a real application path.

There is a subtlety with restic’s change detection. restic uses file metadata — mtime, ctime, size, inode on Unix — to decide whether a file is likely unchanged, and skips rescanning it. If the canary is rewritten every run with a new timestamp, its metadata changes and it will be rescanned. That is what you want. If you write the canary once and never touch it, restic may treat it as unchanged and the canary stops proving anything about the current run. Rewrite it every time, or use --force on a periodic basis to turn off change detection entirely.

Wiring it into the rotation

The scheduling split matters more than the tooling. Content assertions are fast and cheap; run them on every backup and let a failure page. Integrity assertions are slow and expensive; run them on a subsetted cadence in the background and let a failure open a ticket. A team of five engineers cannot afford a nightly full read-back of a multi-terabyte repository, and it does not need one. It needs the cheap check to be reliable and the expensive check to be regular.

Concretely: the backup job writes the canary, runs the backup, lists the latest snapshot, asserts the canary path and one application path, and exits nonzero on any failure. That exit code is what your monitoring watches. The integrity pass runs separately, on a schedule that covers the repository over time, and reports into a queue rather than a pager. If the integrity pass finds corruption, you have time to act; if the content assertion fails, you are already backing up nothing and need to know now.

For PostgreSQL, the same split applies. Run pg_verifybackup against each base backup as part of the backup job — it is the content-and-checksum check — and keep the full restore rehearsal on a slower cadence. The PostgreSQL documentation is direct that verification is not a substitute for test restores.

What this does not prove

A passing canary and a clean checksum rule out emptiness and corruption. They do not rule out a backup that cannot be brought up: a missing WAL segment, a version mismatch, a restore procedure that references a path that no longer exists. Those failures live in the restore path, and the only way to find them is to restore.

That is why the restore lottery stays a separate ritual. Pick a backup at random, restore it to a scratch environment, and assert that the application starts and the canary is present with the expected checksum. The canary does double duty here: it is the content assertion during backup and the end-to-end assertion during restore. If the restored canary matches the checksum recorded at write time, you have evidence that the bytes survived the full round trip.

If you have not written down the restore steps yet, do that before adding more checks. A verification job that nobody knows how to act on is its own kind of hollow backup. The recovery checklist is the companion artifact to this one: the checks tell you whether to reach for it, and the checklist tells you what to do when they fail.

A checklist for this week

• Write a canary file into each backup scope before the job runs, with a timestamp and a fixed string. Rewrite it every run.
• Add a content assertion to the backup job: list the latest snapshot and fail if the canary path or one application path is missing.
• Confirm the assertion actually fails when it should. Point the job at an empty directory once and watch it exit nonzero.
• Schedule an integrity pass — restic check --read-data-subset or pg_verifybackup — on a cadence you can sustain, and route its failures to a ticket, not a page.
• Keep the restore lottery separate. Restore a random backup, start the application, and compare the restored canary checksum to the recorded value.

The exit code is a statement about the process. The canary and the checksum are statements about the payload. Keep them separate, schedule them differently, and let each fail for its own reason.