Using a Pre-Mortem to Pressure-Test a Database Migration Before the Maintenance Window

A maintenance window is a planned change with a known failure surface. That makes it a good candidate for a pre-mortem: a short exercise in which the team assumes the change has already failed and writes down why. Gary Klein’s description of the technique is blunt about the mechanism — people are reluctant to voice reservations during planning, and a pre-mortem makes it safer for knowledgeable dissenters to speak up (Harvard Business Review, 2007).

For a team of two to fifteen engineers who are their own on-call rotation, the pre-mortem is not a risk register and not a prediction. It is a drill-design tool. Its output should be a set of rehearsable checks with pass/fail results, each tied to one named failure mode. The window gets scheduled when the checks have results, not when the team has promised to be careful.

Frame the exercise before you frame the migration

Klein’s instruction is to assume the project has already failed and then generate plausible reasons. Applied to a PostgreSQL migration, the opening sentence is not “what could go wrong” but “the window is over and it went badly — what happened?” The answers tend to cluster around a small number of surfaces: the backup, the archive, version and extension compatibility, the rollback path, and the people.

Run the session with the people who will actually be on the call. In a small team that is usually everyone who touches the database, plus whoever owns the application that reads from it. One person writes; the writer is not the decision-maker. If the team has no dedicated facilitator, rotate the writing role and keep the decision-maker’s hands off the keyboard so they can listen.

The pre-mortem should be run against the rollback plan as hard as against the forward plan. If the team cannot state in one sentence what “abort” means and who says it, the exercise has found its most important gap before it has found anything else.

Failure mode 1: the backup you would restore from is not the backup you think it is

The first plausible reason the window failed is that the restore did not work. This is the failure mode most worth converting into a drill, because it is cheap to test and expensive to discover at 02:00.

PostgreSQL’s documentation is explicit that recovery from a continuous archive requires a continuous sequence of archived WAL files extending back at least as far as the start time of the backup (PostgreSQL 18, Section 25.3). It also advises setting up and testing the WAL archiving procedure before taking the first base backup. The pre-mortem check that follows is concrete: restore a base backup plus the WAL needed to reach a chosen recovery target into a scratch instance, on hardware that is not the production host, before the window. Record the recovery target, the command used, and the time to completion.

If the team uses pgBackRest, the documentation states that a restore requires the backup files and one or more WAL segments to work correctly (pgBackRest User Guide). That is the minimum the drill must exercise. A restore that only copies files and never starts the cluster has not tested the thing the pre-mortem is worried about.

For file-level backups outside PostgreSQL — configuration directories, application secrets, the runbook repository — the same logic applies. restic’s documentation describes repository integrity checking and snapshot listing, but integrity checking is not the same as a restore drill. The pre-mortem should name which restic snapshots the migration depends on and who will restore one to a scratch path before the window.

Failure mode 2: WAL archiving silently stopped

The second plausible reason the window failed is that the archive had a gap the team did not notice. PostgreSQL’s documentation is precise about the contract: the archive command must return zero exit status if and only if it succeeds; a nonzero status tells PostgreSQL the file was not archived and it will retry periodically (Section 25.3.1). The same section warns that if archiving falls significantly behind, more data would be lost in a disaster, and pg_wal/ can accumulate not-yet-archived segments until disk space is exhausted.

The pre-mortem check is a gate, not a glance. Before the window, confirm that the most recent archived segment is recent, that the archive command’s exit behavior is what the team believes it is, and that the monitoring that would catch a stalled archiver is actually firing. If the team uses pgBackRest, note that the documentation requires local and remote pgBackRest versions to match exactly, and that a mismatch prevents WAL archiving and backups from functioning until the versions match (pgBackRest User Guide). A version skew introduced during migration preparation is exactly the kind of quiet failure the pre-mortem exists to surface.

One more check that belongs here: WAL archiving does not restore changes to postgresql.conf, pg_hba.conf, or pg_ident.conf, because those are edited manually rather than through SQL (Section 25.3). If the migration changes any of those files, the pre-mortem should name where the new versions are stored and how they get onto the restored host.

Failure mode 3: version and extension incompatibility

The third plausible reason the window failed is that the target version would not accept something the source version allowed. PostgreSQL’s documentation notes that for major releases the internal data storage format is subject to change, that dump-and-restore is the traditional method, that pg_upgrade is faster, and that replication methods are also available (Section 18.6). The same section recommends that cautious users test client applications on the new major version before switching over fully, and suggests concurrent installations of old and new versions.

Extension compatibility is where this failure mode usually bites. Azure Database for PostgreSQL flexible server documents upgrade validation checks that evaluate compatibility and configuration without initiating the upgrade, and states that these checks do not modify the server version, trigger downtime, or restart the server (Microsoft Learn). The same documentation states that in-place major version upgrades do not support all PostgreSQL extensions, and that the upgrade fails during precheck if a blocked extension is present on an affected upgrade path. That is a useful model even for teams not on Azure: enumerate every extension, every version-specific blocker, and every object that depends on an extension, and resolve them before the window rather than during it.

The pre-mortem check is an inventory with a result per row: extension name, installed version, target-version support, owner, and disposition. A row without a disposition is a check that has not passed.

Failure mode 4: no tested rollback

The fourth plausible reason the window failed is that the team could not get back. Azure’s documentation is direct about the shape of this problem: after a successful in-place major version upgrade there are no automated ways to revert to the earlier version, though point-in-time recovery to a time before the upgrade can restore the previous version on a new server (Microsoft Learn). Self-managed PostgreSQL has the same property in different clothing: the rollback is a restore, and a restore is a procedure that has to be rehearsed.

The pre-mortem should force the team to write the abort criterion in one sentence and name the person who calls it. It should also force a second sentence: what the rollback actually is — a restore to a new host, a DNS switch back to the old primary, a logical replication reversal — and how long it takes. If the answer is “we would figure it out,” the pre-mortem has found a gap that no amount of scheduling discipline will close.

Failure mode 5: the on-call rotation is the same two people

The fifth plausible reason the window failed is staffing. In a team that is its own on-call rotation, the migration window competes with whatever else pages that night. The pre-mortem should include a staffing failure mode: who is awake, who is the decision-maker, who is the scribe, and what happens if the window overruns into the next working day.

This is a planning argument, not a claim about any specific team. The point is that the pre-mortem is the right place to notice that the person who knows the extension inventory is also the person who would be paged for an unrelated incident at the same hour. Assign roles in advance: operator, scribe, decision-maker. Write a stop-work condition — a time, a metric, or an event — that ends the window regardless of progress.

Convert each failure mode into a check with an owner and a result

The pre-mortem is only useful if it changes what happens before the window. The conversion is mechanical: every plausible failure becomes a check, every check gets an owner and a deadline, and every check returns a pass, a fail, or a not-run. The window is scheduled when every check has a result. A promise to be careful is not a result.

A hypothetical example, clearly labeled as such: a two-person team plans a PostgreSQL major-version window. The pre-mortem lists “the extension we forgot is blocked on the target version” and “the archive command has been failing quietly for a week.” Each becomes a check with an owner and a deadline before the window. The first check is an extension inventory reconciled against the target version’s release notes. The second is a review of the archiver’s recent history and the monitoring that would have caught a stall. The window is scheduled only when both checks return a result. If either fails, the window moves; that is the pre-mortem doing its job.

This is also where the pre-mortem connects to the recovery checklist the team should already have. Writing the recovery checklist before you need it is the same discipline applied to a different artifact: Write the Recovery Checklist Before You Need It describes the checklist as something you build and test in advance, not something you assemble during an incident. The pre-mortem is the moment to confirm that the checklist exists, that it names the current backup repository, and that someone has run it end to end.

After the window: read the postmortem against the pre-mortem

Google’s SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause or causes, and the follow-up actions to prevent recurrence (Site Reliability Engineering, Chapter 15). It also states that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary, and that a blameless postmortem focuses on contributing causes without indicting any individual or team.

The pre-mortem gives the postmortem a comparison set. After the window, the team can ask which predicted failures happened, which did not, and which surprises were absent from the list. Predicted failures that did not occur are as informative as unpredicted ones, because they show where the team’s model of the system is miscalibrated. A pre-mortem that predicted an archive gap and found none is not wasted; it is evidence that the archive check worked, and it should be recorded as such.

The postmortem should also record whether the abort criterion was used, and if not, why. A window that overran because no one was willing to call the abort is a finding about the plan, not about the people.

What the pre-mortem does not do

A pre-mortem reduces surprise. It does not eliminate downtime, and it is not a substitute for a tested restore. It can schedule the restore drill and name its owner, but the drill itself is the evidence. The pre-mortem only ensures the drill happens before the window rather than during it.

It also does not replace the platform’s own validation. If the team runs on a managed service that offers pre-upgrade validation checks, those checks are the authoritative source for that platform’s blockers, and the pre-mortem should consume their output rather than duplicate it. If the team runs self-managed PostgreSQL, the equivalent is the extension inventory, the version-specific release notes, and a restore drill against the actual backup repository.

The useful claim is narrow: a pre-mortem converts a list of worries into a list of checks, and a list of checks with results is a better basis for scheduling a maintenance window than a list of worries. That is enough.

FAQ

How long should a pre-mortem for a database migration take?

Long enough to enumerate the failure modes and assign owners, short enough that the people who will be on the call actually attend. For a small team, the session is usually one meeting plus asynchronous follow-up on the checks. The checks themselves take longer than the meeting; the meeting’s job is to produce them.

Who should facilitate if the team has no dedicated SRE?

Rotate the writing role and keep the decision-maker out of the scribe seat. The facilitator’s job is to keep the exercise in the past tense — “the window failed, why” — rather than letting it drift into a forward-looking risk review. If the team is two people, one writes and one talks, and they swap for the next migration.

Does a pre-mortem replace a restore drill?

No. The pre-mortem can schedule the drill and name its owner, but the drill is the evidence. PostgreSQL’s documentation advises testing the WAL archiving procedure before taking the first base backup; the pre-mortem is the mechanism that ensures the test happens before the window rather than during it.

What if the pre-mortem finds a failure mode the team cannot fix before the window?

Then the window moves, or the scope changes, or the team accepts the risk explicitly and records the acceptance. The pre-mortem’s value is that the acceptance is a decision rather than an oversight. A check that returns “fail, accepted by [name] on [date]” is a result; a check that returns nothing is not.

How does this differ from a standard change-management review?

A change review asks whether the change is approved. A pre-mortem asks what would make it fail and whether the team has rehearsed the answer. The two can share a meeting, but the pre-mortem’s output is a set of checks with results, not an approval.