Rehearsing a Cross-Region Restore into a Fresh AWS Account Before You Actually Need Disaster Recovery

by ·Comments Off on Rehearsing a Cross-Region Restore into a Fresh AWS Account Before You Actually Need Disaster Recovery

Why a fresh account is the right rehearsal target

A restore drill that runs inside the production account is a weaker test than one that runs in a fresh account. The production account silently supplies IAM roles, VPC defaults, DNS, and credentials that a real disaster may not. When you rehearse into a new account, you are testing the parts of recovery that are usually invisible until they fail: identity, network reachability, key material, and the operational knowledge of the person holding the pager at 03:00.

This is a drill-first guide for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated SRE department. The target is a PostgreSQL cluster restored from a pgBackRest repository into a fresh AWS account in a different region. The same reasoning applies to restic repositories and to bare-metal targets, with the tool-specific mechanics called out where they differ.

The argument is simple: treat the fresh account as a hostile environment. Assume nothing from production is present. Rebuild the backup path from scratch, verify the restore against the backup tool’s own documented mechanics, and record what the drill did not prove.

Define success before touching anything

A restore command that exits zero is not a successful drill. The pass criterion should be a verification query against the restored database: the cluster is out of recovery, the recovery target timestamp is known, and row counts on a few named tables match expectations within a tolerance you set in advance.

Write the criterion down before the drill starts. A workable example:

  • A PostgreSQL instance is running in the fresh account and region.
  • pg_is_in_recovery() returns false.
  • The recovery target timestamp actually reached is recorded.
  • Row counts for three named tables are within a stated tolerance of the source at the chosen target time.
  • The restore host’s pgBackRest version matches the version that wrote the archive.

Everything else — DNS cutover, application reconfiguration, credential rotation — is out of scope for this drill and belongs in the decision record as unproven.

Step 1 — Provision the target account deliberately

Create the recovery account as a member account in your AWS organization rather than as a standalone account. AWS Organizations automatically creates the IAM role OrganizationAccountAccessRole in the member account, which enables users and roles in the management account to exercise full administrative control over it. That role is the documented bootstrap path; decide in advance whether you will use it or replace it with a narrower break-glass role.

Two consequences matter for a drill. First, policies attached to the organization root immediately apply to all users and roles in the newly created account. If your root has service control policies that restrict regions or services, the recovery account inherits them. That is usually desirable, but it is a constraint you should know about before the drill, not during it. Second, the account is created in the root of the organization and can be moved between organizational units afterward; place it where its SCPs match the recovery intent.

Record the account ID, the region, and the role name in the drill’s decision record. The account is an artifact of the drill, not a permanent fixture, and it should be tearable down without ambiguity.

Step 2 — Rebuild the backup path from scratch

The restore host in the fresh account must reach the repository. For pgBackRest that means network reachability to the repository host or object store, the repository password or encryption key, and a pgBackRest binary of the correct version.

Version matching is not optional. The pgBackRest user guide states that the local and remote pgBackRest versions must match exactly and should be upgraded together; if there is a mismatch, WAL archiving and backups will not function until the versions match. A restore host running a newer pgBackRest than the repository host is a failure mode worth rehearsing deliberately rather than discovering during an incident.

Stage the repository password or encryption key outside the production account. If the only copy lives in a production secrets manager that the fresh account cannot read, the drill has already found a real gap. The same reasoning applies to restic: the repository password is required to open the repository, and a copy that exists only in production is not a recovery asset.

For restic repositories, note that the copy command reads and uploads the entire snapshot because the source and destination repositories use different encryption keys, which may incur higher bandwidth usage and costs than normal backup runs. It also does not re-chunk files, which may break deduplication between copied files and files already in the destination repository. If your drill includes moving a restic repository into the fresh account, budget for the full transfer and do not assume deduplication will hold across the copy.

Step 3 — Choose the restore mode that matches the drill

pgBackRest supports several restore modes, and the drill should name which one it is testing.

  • Full restore to latest. For a standard non-delta restore, the data directory must be empty. This is the simplest mode and the right first drill.
  • Specific backup set. By default pgBackRest restores the most recent backup; --set restores a specific backup label. Use this when you want to prove that an older backup is still restorable.
  • Delta restore. Compares checksums against the backup manifest and replaces only files that differ. Faster for large clusters, but it assumes a data directory already exists, so it is a second drill, not a first.
  • Point-in-time recovery. Replays WAL to a target timestamp. In pgBackRest, --target-action=promote causes PostgreSQL to promote to a primary after reaching the recovery target; without it, the cluster pauses in recovery at the target point and waits for manual intervention. A drill that omits the promote action will appear to hang, which is a useful thing to see once and then never again.

PostgreSQL’s own continuous-archiving documentation is worth reading alongside the tool docs. It notes that pg_dump and pg_dumpall do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. If your recovery plan depends on logical dumps, the drill will not exercise WAL replay, and the pass criterion should say so.

The same documentation notes that WAL archiving will not restore changes made to configuration files such as postgresql.conf, pg_hba.conf, and pg_ident.conf, since those are edited manually rather than through SQL operations. In a fresh account, those files must come from somewhere else — a configuration repository, a secrets store, or a documented reconstruction step. The drill should prove that the somewhere else exists.

Step 4 — Handle what breaks in a fresh account

Three categories of failure are common when restoring into a different host or directory layout, and all three are documented pgBackRest behaviors rather than surprises.

Tablespace paths

When restoring to a different host or directory layout, tablespace paths from the original cluster may not exist. Use --tablespace-map to redirect individual tablespaces, or --tablespace-map-all to redirect all tablespaces to a single directory. Rehearse the flags before the drill, not during it.

Symlinked directories

pgBackRest records symbolic links inside the data directory in the backup manifest and recreates them on restore. If pg_wal points to a separate disk on production, the restore host needs a corresponding path or a --link-map to redirect it. A restore that fails because /mnt/wal-disk does not exist in the fresh account is a configuration gap, not a backup failure.

File ownership

For a non-root restore, all restored files belong to the user and group running pgBackRest. If existing files have different ownership and the executing user cannot change it, the restore fails. For a root restore, pgBackRest recreates the ownership recorded in the backup manifest, but only user and group names are stored, not numeric IDs, so the same names must exist on the restore host. Decide which model the drill uses and make the user and group names match.

Step 5 — Verify against the database, not the exit code

Connect to the restored cluster and run the checks you wrote down in step zero. A minimal sequence:

SELECT now(), pg_is_in_recovery();
SELECT schemaname, tablename, n_live_tup
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC
LIMIT 20;

Record the recovery target timestamp actually reached. If the drill used PITR, compare it to the intended target. If the drill restored to latest, record the timestamp of the most recent WAL segment replayed. Either way, the number goes in the decision record.

If the drill uses the Crunchy Postgres Operator, the documentation describes bootstrapping a new cluster from a pgBackRest repository via spec.dataSource as the recommended way to stand up a clone or replacement cluster from an off-site backup. The same documentation warns that if the original primary cluster is still running when a standby is promoted, you will have two writable clusters pointing at the same data, and advises ensuring the original primary is offline or network-isolated before promotion. In a fresh account, the original primary is not reachable by default, which is one of the reasons the fresh account is a good rehearsal target — but the split-brain warning still belongs in the runbook for the real event.

Step 6 — Tear down and write the decision record

The drill is not finished until the decision record exists. Put it where the runbooks live, not in a chat thread. A useful record answers four questions:

  • What did the drill prove? Name the restore mode, the backup set or target timestamp, and the verification queries that passed.
  • What did it not prove? DNS cutover, application reconfiguration, credential rotation, and any service outside PostgreSQL are the usual omissions. Say so explicitly.
  • Which runbook step was wrong? If the drill required a flag that the runbook does not mention, the runbook is the defect.
  • What is the next drill’s target? A delta restore, an older backup set, a PITR to a named restore point, or a restic repository opened from the fresh account.

This is the same discipline as a blameless postmortem, applied before the incident. The recovery checklist is the natural companion artifact: the drill tests the checklist, and the checklist records what the drill found.

A short pre-mortem for the next drill

Before scheduling the next drill, ask what would make it fail silently. A few prompts that have earned their place:

  • If the repository password were rotated between drills, would the fresh account still open the repository?
  • If the pgBackRest version on the restore host drifted, would the drill detect the mismatch before the restore command, or after?
  • If the target timestamp were earlier than the selected backup’s stop time, would the operator recognize the failure as a target-selection problem rather than a corrupt backup?
  • If the person running the drill were not the person who wrote the runbook, would the runbook still be sufficient?
  • If the fresh account’s SCPs blocked a service the restore needs, would the error message name the SCP or the service?

None of these require new tooling. They require that the drill be treated as a test of the team’s knowledge, not only of the backup.

FAQ

Why not rehearse in the production account with a separate VPC?

Because the production account supplies IAM roles, DNS, and credentials that a real disaster may not. A separate VPC inside the same account tests network configuration but not account-level identity and key access. The fresh account is the harder test, and the harder test is the one worth scheduling.

Do we need a full copy of the repository in the fresh account?

Not necessarily. The restore host can read from the existing repository if network reachability and credentials allow it. What must exist in the fresh account is the ability to authenticate to the repository and the pgBackRest binary of the matching version. For restic, the repository password is the equivalent requirement.

How often should the drill run?

Often enough that the runbook is current and the on-call rotation has run it at least once. For a small team, once per quarter per critical service is a reasonable starting cadence; the decision record from each drill should state whether the cadence still fits.

What about RDS or S3 backups?

This drill is written around pgBackRest and PostgreSQL WAL because those mechanics are documented in the sources cited here. If your recovery path uses managed snapshot services, the drill should be rewritten around those services’ documented restore procedures, and the pass criterion should still be a verification query rather than a console success message.

Sources