How to Rotate Backup Encryption Keys Without Losing Access to Old Archives

by ·Comments Off on How to Rotate Backup Encryption Keys Without Losing Access to Old Archives

The question behind the question

“Rotate the backup key” is ambiguous, and the ambiguity is where teams get hurt. In most backup stacks there are at least two layers of encryption: an outer credential that unlocks the repository or wraps metadata, and inner keys that encrypt the actual archive contents. Rotating the outer layer is a credential-lifecycle operation. Rotating the inner layer is a re-encryption project. Confusing the two produces either a false sense of security (you changed a password and believe the archives are re-encrypted) or an unnecessary rewrite of terabytes of immutable history.

For a 2–15 engineer team that is its own on-call rotation, the practical scope is narrower than vendor marketing suggests: issue a new access credential, prove old archives still open, then revoke the old credential. The dangerous failure mode is ordering — removing the old key before a verified restore from an archive created under it. This article maps what each common tool actually protects, then gives a drill-first procedure that keeps old archives readable.

Map the layers before touching anything

Write down, for each backup system you run, which key protects which artifact. The answer differs by tool, and the differences matter.

restic: repository password keys

restic’s key command “allows you to set multiple access keys or passwords per repository” (restic-key(1)). That is the whole rotation model: a repository can have several access credentials, each of which can decrypt the repository’s master key material. Adding a key does not re-encrypt any snapshot data. Removing a key revokes that credential’s ability to open the repository; it does not change the ciphertext of existing snapshots.

One command deserves special caution. restic key passwd “creates a new key, validates the key and removes the old key ID” (restic-key-passwd(1)). It is a combined add-and-remove in a single step. If the new password is not yet stored somewhere your recovery process can reach, you have just removed the working credential. Prefer the explicit add, verify, remove sequence over passwd when the repository is the only copy of production history.

pgBackRest: repository cipher passphrase and subkeys

pgBackRest “will encrypt the repository based on a user-provided password” (pgBackRest User Guide). The configuration option is repo1-cipher-pass, paired with repo1-cipher-type (pgBackRest Configuration Reference). What that passphrase actually protects is narrower than people assume. As the pgBackRest maintainer explained in a 2019 issue thread: “cipher-pass is only used to encrypt backup.info and archive-info. backup.info contains a subkey that encrypts backup.manifest and each backup set has its own subkey” (pgbackrest#680).

That hierarchy is the reason rotation is tractable. The master passphrase wraps small metadata files. The metadata files carry the subkeys that encrypt manifests and WAL. Each backup set already has its own subkey, and per the same thread, the manifest key is rotated on every full backup. The archive key that encrypts WAL is not currently rotated, and a stanza-rekey command is an open feature request, not a shipped feature (pgbackrest#2814). Do not plan around a command that does not exist in your installed version.

The documented-by-maintainer rotation path for the master passphrase is to re-encrypt archive.info and backup.info under a new key. The maintainer described doing this with the openssl command-line tool because the encrypted files are OpenSSL-compatible (pgbackrest#680). That is a manual, high-consequence operation. Treat it as a change with a rollback plan and a restore test, not a config edit.

age: recipients on individual files

age has no repository and no master key. Each file is encrypted to one or more recipients. “Files can be encrypted to multiple recipients by repeating -r/--recipient. Every recipient will be able to decrypt the file” (age README). Identity files “contain one or more secret keys … Multiple key files can be provided, and any unused ones will be ignored” (same source).

This means rotation for age-encrypted archives is a forward-looking operation. New files go to new recipients. Old files remain decryptable only by the identities they were encrypted to. There is no in-place re-wrap. If policy requires old files to be readable only by the new recipient, you must decrypt and re-encrypt them — a real data operation with real cost, not a key-management step.

AWS KMS: key material versions, not data re-encryption

If your backups are wrapped by a KMS key, the rotation semantics are explicit: “Key rotation has no effect on the data that the KMS key protects. It does not rotate the data keys that the KMS key generated or re-encrypt any data protected by the KMS key” (AWS KMS Developer Guide). Decryption is transparent: “When you use the rotated KMS key to decrypt ciphertext, AWS KMS uses the same version of the key material that was used to encrypt it” (same source). AWS KMS retains all key material for a KMS key with AWS_KMS origin even if rotation is disabled, and deletes it only when the key itself is deleted (same source).

The operational consequence: rotating a KMS key does not make old ciphertext unreadable, and deleting the KMS key does. Schedule deletion with the same care you would give to deleting the only copy of a password.

The safe order of operations

Across all four tools, the sequence that preserves access is the same:

  1. Add the new credential or recipient alongside the old one.
  2. Verify that the new credential opens the repository and that the old credential still opens archives created before the change.
  3. Restore a real artifact from an archive created before rotation — not a listing, not a metadata read.
  4. Remove the old credential only after step 3 succeeds and the new credential is stored where your recovery process can reach it.

Steps 1 and 4 are reversible. Step 3 is the one that catches mistakes. A rotation that ends with a successful restic snapshots or pgbackrest info has proven that metadata is readable. It has not proven that the data behind the metadata is recoverable under the new credential arrangement.

restic walkthrough

The exact subcommand syntax for key add and key remove varies by restic version; check restic key --help on the installed binary before scripting. The shape of the operation is stable:

# 1. Add a new access key, reading the new password from a file
restic -r s3:s3.amazonaws.com/example-backups key add \
  --new-password-file /run/secrets/restic-new-password

# 2. List keys and record the new key ID
restic -r s3:s3.amazonaws.com/example-backups key list

# 3. Verify the new credential opens the repository
RESTIC_PASSWORD_FILE=/run/secrets/restic-new-password \
  restic -r s3:s3.amazonaws.com/example-backups snapshots --latest 1

# 4. Restore a file from a snapshot created before rotation
RESTIC_PASSWORD_FILE=/run/secrets/restic-new-password \
  restic -r s3:s3.amazonaws.com/example-backups restore latest \
  --target /tmp/restore-lottery --include /etc/example.conf

# 5. Only after step 4 succeeds, remove the old key by ID
restic -r s3:s3.amazonaws.com/example-backups key remove <old-key-id>

Two notes. First, key passwd collapses steps 1 and 5 into one command and removes the old key ID as part of its operation (restic-key-passwd(1)); avoid it for the primary repository unless you have already stored the new password in your break-glass path. Second, removing a key does not shrink the repository or change snapshot ciphertext. The old snapshots remain encrypted under the repository’s master key material, which the remaining access keys can unwrap. Removing the last access key you control is the only way to lose access.

pgBackRest walkthrough

pgBackRest does not ship a supported rekey command as of the 2.60/2.61 milestone discussion (pgbackrest#2814). The master passphrase rotation is a manual re-encryption of archive.info and backup.info. The maintainer’s description of the mechanism is the authoritative starting point (pgbackrest#680); the exact openssl invocation depends on the cipher type configured in repo1-cipher-type and should be rehearsed against a copy of the repository, not the live one.

Before any manual re-encryption, confirm what you are changing:

  • repo1-cipher-pass wraps backup.info and archive.info.
  • backup.info carries the manifest subkey and per-backup-set subkeys.
  • archive.info carries the archive key used for WAL.
  • Historical WAL and manifests stay under their prior subkeys. Re-wrapping the info files does not re-encrypt them.

That last point is the one to write into the runbook. A reader who assumes rotation re-encrypts WAL will make wrong decisions about retention and about whether a compromised passphrase still exposes old archives. It does, until the subkeys themselves are rotated, and the archive key is not currently rotated by the tool.

PostgreSQL’s own documentation reinforces why the WAL chain must stay intact: “To recover successfully using continuous archiving … you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup” (PostgreSQL 18, Continuous Archiving and PITR). Any rotation procedure that interrupts WAL archiving or changes the archive identity mid-stream risks a gap. The 2019 issue thread documents exactly this hazard when a second repository was introduced with a new key and retention purged WAL prior to the first backup in the new repo (pgbackrest#680). The lesson is not that rotation is unsafe; it is that rotation and repository migration are different operations and must not be combined in one change window.

age and file-level archives

For archives encrypted directly with age, rotation is a recipient change, not a repository change. Add the new recipient to the encryption command for new files, keep the old identity available for decryption, and decide explicitly whether old files need re-encryption.

# New files go to both recipients during the transition
age -r age1newrecipient... -r age1oldrecipient... \
  -o backup-2026-10.tar.gz.age backup-2026-10.tar.gz

# Old files decrypt with the old identity, which is still on disk
age --decrypt -i old-identity.txt backup-2025-03.tar.gz.age > backup-2025-03.tar.gz

“Every recipient will be able to decrypt the file” (age README), so a dual-recipient transition period is safe and reversible. The irreversible step is deleting the old identity before you have confirmed that no archive you still need was encrypted only to it. If policy requires old archives to be readable only by the new recipient, that is a decrypt-and-re-encrypt job with storage and time costs; budget it as such rather than treating it as part of key rotation.

A restore-lottery drill that proves the rotation worked

The verification step in the procedure above should be a real restore, not a metadata query. A restore lottery picks a random archive from before the rotation date and restores it to a scratch location. The drill is small enough to run during an on-call shift and specific enough to catch the failure modes that matter.

  1. Pick a snapshot or backup set created at least one retention cycle before the rotation date.
  2. Restore it using only the new credential. Do not fall back to the old one.
  3. Compare a checksum or a known row count against the expected value recorded at backup time.
  4. Record the archive ID, the restore timestamp, and the credential used in the decision record.

If step 2 fails, the rotation is not complete regardless of what key list or info reports. The recovery checklist article at Write the Recovery Checklist Before You Need It is a useful companion for structuring the drill so it can be executed by whoever is on call, not only by the person who designed it.

Runbook and decision-record notes

Record, at minimum: the key ID or key version, the rotation date, the archive IDs used in the restore lottery, and the name of the person who holds break-glass access to the new credential. The next engineer to answer “which key opens the 2024 archive?” should not have to reconstruct that from chat history. For pgBackRest, note explicitly that the archive key and manifest key were not rotated by the master-passphrase change, so the decision record does not overstate what was accomplished. For restic, note which key IDs remain and which were removed. For age, note which identities are still required to read retained archives.

A rotation that requires a second person who is asleep is a rotation that gets deferred indefinitely. Keep the runbook short enough to execute solo, and keep the break-glass credential in a location the on-call engineer can reach without a meeting.

What not to do

  • Do not remove the old key before a verified restore from an archive created under it. restic key passwd does both in one step; understand that before running it.
  • Do not assume rotation re-encrypts stored data. AWS KMS states it plainly (AWS KMS Developer Guide), and pgBackRest’s subkey hierarchy has the same property.
  • Do not combine key rotation with repository migration or retention changes in one window. The 2019 pgBackRest thread shows how a new repository plus a new key plus retention produced a WAL gap (pgbackrest#680).
  • Do not plan around a stanza-rekey command until it ships in the version you run. It is an open feature request (pgbackrest#2814).
  • Do not delete a KMS key to “rotate” it. Deletion removes the key material; rotation does not (AWS KMS Developer Guide).

FAQ

Does removing a restic key make old snapshots unreadable?

No. Removing an access key revokes that credential’s ability to open the repository. The snapshots remain encrypted under the repository’s key material, which the remaining access keys can unwrap. Access is lost only when no remaining key can open the repository.

Does rotating the pgBackRest repo1-cipher-pass re-encrypt WAL?

No. The passphrase wraps backup.info and archive.info. WAL is encrypted under the archive key carried in archive.info, and that key is not currently rotated by the tool (pgbackrest#2814). Historical WAL stays under its prior key.

Can I decrypt an age file with an identity that is not the current recipient?

Only if that identity was a recipient when the file was encrypted. age has no re-wrap operation; old files keep the recipients they were encrypted to. Keep old identities available for as long as you retain archives encrypted to them.

Does AWS KMS rotation break decryption of old backups?

No. AWS KMS automatically uses the key material version that encrypted the ciphertext when decrypting (AWS KMS Developer Guide). Deleting the KMS key is the operation that removes that ability.

How often should we rotate?

That is a policy decision, not a technical constant. The technical constraint is that rotation must be followed by a verified restore from a pre-rotation archive before the old credential is retired. Choose an interval your team can actually execute end to end, including the restore lottery.