A backup job that exits zero every night answers one question: did the upload complete. It does not answer whether the repository is internally consistent, whether a restore would succeed, or whether the snapshot contains the data you believe it does. For a team of two to fifteen engineers who are their own on-call rotation, that gap is the whole risk. The nightly cron is green; the restore path has never been exercised.
This article is about closing that gap without decrypting production data and without standing up a second production environment. The premise needs one correction up front: you cannot validate an encrypted backup without ever decrypting anything. restic’s check and restore both require the repository key. What you can do is keep every validation step off production paths, off production credentials, and off production data — and separate the work into three checks that can be scheduled independently.
Three questions, three checks
“Is the backup valid?” is not one question. It decomposes into:
- Integrity. Is the repository internally consistent, and are the stored data files unmodified?
- Restore-path proof. Can we actually restore a snapshot to a target we control?
- Coverage. Does the snapshot contain the hosts, paths, and data we think it does?
Each check has a different cost and a different failure mode. Splitting them lets a small team run the cheap ones often and the expensive ones on a rotation, rather than trying to do everything in one long maintenance window nobody wants to own.
Listing is not validating
restic’s snapshots command lists snapshot metadata: ID, date, host, tags, directory, and — for snapshots created with restic 0.17.0 or later — size. The ls command lists the files inside a specific snapshot, and the special snapshot ID latest selects the most recent one. Both are useful for coverage checks. Neither reads the repository’s data files.
A green backup cron plus a successful restic snapshots is the weakest possible evidence. It proves the repository index is readable. It does not prove the packs behind that index are intact.
Layer one: integrity
restic’s own documentation is explicit about the default: “By default, check command does not check that repository data files are unmodified.” The check command verifies the repository’s structure — indexes, snapshots, trees, and blobs — but reading the actual data files requires an additional flag.
Two flags change that:
--read-datareads all repository data files.--read-data-subset=n/tdivides all repository data files into t roughly equal groups and checks only group n.
The subset flag is the affordance that makes integrity checking practical for a lean team. A full --read-data pass on a large repository is a long-running job that competes with backup windows and on-call attention. Spreading it across t scheduled invocations amortizes the cost: run --read-data-subset=1/5 this week, 2/5 next week, and so on, and every data file is read once per cycle. The restic documentation shows exactly this pattern across five invocations.
Two operational notes from the same documentation. First, restic advises running restic check after prune, so that damage to internal data structures surfaces immediately rather than at the next drill. Second, the check command is presented as the regular mechanism for confirming that backup data is consistent and its integrity is unharmed — not a one-time setup step.
What this layer does not prove: that a restore will succeed end to end, that the snapshot contains the right data, or that the repository is tamper-evident against an attacker with write access. It proves the stored bytes are readable and consistent with the repository’s own structures. That is a real property, and it is not the same as recoverability.
Layer two: restore-path proof
The only check that proves the restore path is a restore. The discipline is in where it lands.
Restore to an isolated scratch target with its own credentials — a separate host, a separate bucket, a separate filesystem that shares no identity with production. The point of the drill is to exercise the restore path, not to test your ability to be careful under pressure. If the drill target can reach production paths, the drill is one typo away from an incident.
A drill script shape, clearly hypothetical and meant to be adapted rather than copied:
# Runs on an isolated drill host with its own repository credentials.
# Never point RESTIC_REPOSITORY at a production path.
export RESTIC_REPOSITORY="s3:s3.example-region.amazonaws.com/drill-restic"
export RESTIC_PASSWORD_FILE="/etc/restic/drill-password"
SNAPSHOT_ID="$(restic snapshots --host app-01 --json | jq -r '.[-1].short_id')"
TARGET="/srv/drill/restore-${SNAPSHOT_ID}"
mkdir -p "${TARGET}"
restic restore "${SNAPSHOT_ID}" --target "${TARGET}"
# Assertions: the restore must produce the files you expect.
test -f "${TARGET}/srv/app/config/production.yaml"
test -d "${TARGET}/srv/app/uploads"
# Record the result for the runbook.
echo "drill ok snapshot=${SNAPSHOT_ID} target=${TARGET} at=$(date -u +%FT%TZ)"The assertions matter more than the restore. A restore that completes but produces an empty directory is a failed drill. Pick a small set of files and directories whose presence is non-negotiable — the application config, the upload directory, the database dump if you take one — and assert on them explicitly.
One cost note from restic’s documentation: the copy command must download and upload the entire snapshot because source and destination repositories use different encryption keys. If your drill strategy involves copying snapshots to a separate drill repository, budget for that bandwidth. Restoring directly from the offsite repository to the drill host avoids the copy entirely and is usually the better shape for a small team.
Layer three: coverage
Coverage is the cheapest check and the one most often skipped. It answers: does the snapshot contain the hosts and paths we believe it does?
restic’s snapshots command filters by --path and --host, and combines filters. A coverage check is a short script that asserts, for each host in your inventory, that a snapshot exists within the expected window and includes the expected paths. Run it daily. It catches the failure mode where a backup job silently stops including a directory after a config change — a failure that integrity checks and restore drills will not catch if the drill happens to target a snapshot from before the change.
Coverage is also where you catch the inverse: a snapshot that includes more than you intended. If a path was added to the backup set by accident, coverage assertions on the expected set will not flag it, but a periodic review of restic ls --long latest output against the intended inventory will.
The PostgreSQL case: a file restore is half the story
For teams running PostgreSQL with continuous archiving, a file-level restore drill proves the base backup is readable. It does not prove point-in-time recovery works. The PostgreSQL documentation is direct: “To recover successfully using continuous archiving … you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup.” The unit of recovery is the base backup plus the WAL chain from that backup forward, not the base backup alone.
That means the PostgreSQL drill has its own shape: restore the base backup to an isolated target, configure recovery to replay WAL to a chosen target time, start the server, and assert on the state of a known table. The PostgreSQL documentation also advises setting up and testing the WAL archiving procedure before taking the first base backup — the drill is not an afterthought to the archiving setup, it is part of it.
Two related cautions from the same documentation. pg_dump and pg_dumpall do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution; a logical dump drill does not substitute for a PITR drill. And WAL archiving does not capture changes to postgresql.conf, pg_hba.conf, or pg_ident.conf, so those files need to be in your regular file-system backup set and covered by the coverage check.
pgBackRest’s verify command is a useful complement, not a replacement. It checks database and archive files and reports counts of missing, checksum-invalid, and size-invalid files per backup — the output shape is visible in the project’s issue tracker. But the --set option to scope verification to a single backup is marked “FOR INTERNAL USE ONLY. DO NOT USE IN PRODUCTION” in its help text, per the same issue. Treat verify as an integrity layer and keep the PITR rehearsal as a separate drill.
What about age-encrypted files?
If your offsite layer wraps artifacts in age, the format gives you a structural check that does not require the identity. An age file is a textual header carrying a per-file 128-bit symmetric file key, followed by a binary payload encrypted with that key. The header ends with an HMAC-SHA-256 over the header itself, keyed from the file key. The payload is split into 64 KiB chunks, each encrypted with ChaCha20-Poly1305, and streaming decryption must signal an error if the end of file is reached without a successfully decrypted final chunk.
What that buys you: a truncated or bit-flipped age file will fail to decrypt, and the header MAC will fail if the header was modified. What it does not buy you: a repository-level tamper-evidence guarantee. The age specification describes the file format, not a threat model for an offsite store where an attacker has write access. If tamper-evidence matters, that is a separate control — object versioning, immutability, or an append-only repository configuration — not a property of the encryption format.
What object storage can and cannot tell you
S3-compatible stores expose checksums over object bytes at rest. Amazon S3 documents a Compute checksum operation in S3 Batch Operations that calculates checksums for a list of objects without restoring or downloading them, and produces an integrity report. That is a real and useful control: it tells you the bytes S3 holds match the bytes S3 recorded at upload.
It does not tell you the restic repository is logically consistent. Object-level checksums verify storage integrity; restic’s check verifies repository structure. They are different properties, and a team that runs only one of them has a gap. Run both, and label them correctly in the runbook so the next on-call engineer knows which failure each one catches.
Make the drill fail on purpose
A drill that has never failed is a drill whose assertions are too weak. Periodically, in a non-production test repository, corrupt or truncate an artifact deliberately and confirm the drill catches it. restic’s documentation includes exactly this demonstration: appending bytes to an index file causes restic check to report a ciphertext verification failure, and a restore of the affected snapshot fails with the same error. Reproduce that in your own test repository once a quarter. If the drill passes on a deliberately corrupted repository, the drill is not testing what you think it is.
Record the evidence, not the Slack message
The artifact of a drill is inherited evidence. Record, for each run: date, snapshot ID, subset index (for integrity runs), restore target, assertions executed, and outcome. Put it in the same runbook or decision record the on-call rotation reads. A drill result that lives only in a Slack thread is lost at the next rotation handoff.
This is the same discipline described in Write the Recovery Checklist Before You Need It: the checklist is written when the pressure is low so that it can be followed when the pressure is high. The drill record is the checklist’s evidence trail.
A minimal drill checklist for a small team
Adapt the cadence to your repository size and your tolerance for long-running jobs. The structure matters more than the specific numbers.
- Daily: coverage check. Assert that each host in inventory has a snapshot within the expected window and includes the expected paths. Cheap, fast, catches silent scope changes.
- Weekly:
restic checkwithout--read-data, plusrestic checkafter anyprune. Catches structural damage early. - Rotating:
restic check --read-data-subset=n/t, one subset per scheduled run, so every data file is read once per cycle. Choose t so the cycle fits your risk tolerance and the per-run cost fits your team’s attention. - Rotating: restore drill to an isolated scratch target with its own credentials. Assert on a small set of non-negotiable files. Record the result.
- Rotating, PostgreSQL shops: PITR rehearsal — restore base backup, replay WAL to a target time, start the server, assert on known state. This is a separate drill from the file restore.
- Quarterly: deliberate-corruption test in a non-production repository. Confirm the drill catches what it is supposed to catch.
The nightly backup cron stays. It is the thing that produces the artifacts the drill validates. It is not the validation.
FAQ
Can I validate an encrypted backup without decrypting it at all?
Not meaningfully. restic’s check and restore both require the repository key. Object-storage checksums and age header structure can be inspected without the key, but they verify storage integrity and file format, not recoverability. The realistic goal is to keep decryption off production paths and production credentials, not to avoid it entirely.
Does restic check --read-data guarantee the backup is recoverable?
No. It reads the data files and confirms they are consistent with the repository’s structures. It does not prove a restore will produce the files you expect, and it does not prove the snapshot contains the right data. That is why the restore-path and coverage layers exist separately.
Is pgBackRest verify a substitute for a PITR drill?
No. verify checks database and archive files and reports missing, checksum-invalid, and size-invalid counts. It does not replay WAL to a target time and start a server. The PITR rehearsal is a separate exercise.
How often should the restore drill run?
There is no sourced answer. The tradeoff is between the cost of the drill and the age of the evidence you are willing to carry into an incident. Pick a cadence you will actually sustain, record it, and revisit it when the repository grows or the team changes.
What if the drill target is expensive to stand up?
The drill target does not need to be a full production replica. It needs to be isolated, to have its own credentials, and to be able to receive a restore and run assertions. A small instance with enough disk for the snapshot you are testing is sufficient for the restore-path layer. The PITR layer needs a PostgreSQL instance, but it can be a single-node scratch instance with no production connectivity.