What a Blameless Postmortem Should Record When the Trigger Was a One-Line Terraform Typo

by ·Comments Off on What a Blameless Postmortem Should Record When the Trigger Was a One-Line Terraform Typo

What a Blameless Postmortem Should Record When the Trigger Was a One-Line Terraform Typo

A one-line Terraform typo is a trigger, not a root cause. The postmortem that stops at the typo produces a fix as narrow as the mistake and leaves untouched the conditions that made the mistake consequential. For a team of two to fifteen engineers who are their own on-call rotation, the useful postmortem records the plan/apply gap, the state assumptions, the detection lag, and the missing guardrail — then converts those into bounded, owned action items.

This is a writing and analysis discipline, not a tooling purchase. The sources below describe general postmortem practice and Terraform’s documented behavior. They do not document a specific typo-triggered outage, and this article does not invent one. Where an example appears, it is labeled hypothetical.

Start by separating the trigger from the contributing causes

Google’s SRE book defines a postmortem as “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” The same chapter states that a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.”

Applied to a Terraform typo, that means the postmortem has at least two distinct sections:

  • Trigger. The exact change: the file, the resource address, the diff, the commit hash, the plan output that was reviewed, and the apply output that ran. This is a factual record, not a judgment.
  • Contributing causes. The conditions that allowed the change to reach production and to matter: what the plan showed, whether the applied plan matched the reviewed plan, what the team believed about state, how the change was detected, and what the recovery path required.

Google’s example postmortem template keeps these as separate fields — “Root Causes” and “Trigger” — which is a useful structural reminder that the trigger is one input among several. A postmortem that lists only the typo under root causes has collapsed the analysis into a single cause and will generate a single-cause fix.

Record the plan/apply facts, not the typo narrative

Terraform’s documentation is precise about what terraform plan does: it “creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure,” and “the plan command alone does not actually carry out the proposed changes.” The same page warns that “other changes made to the target system in the meantime might cause the final effect of a configuration change to be different than what an earlier speculative plan indicated, so you should always re-check the final non-speculative plan before applying to make sure that it still matches your intent.”

Those documented behaviors give the postmortem a factual frame. Record, with links or pasted excerpts:

  • The speculative plan reviewed in the pull request, if any.
  • The final non-speculative plan shown at apply time, and whether it matched the reviewed plan.
  • Whether the apply used a saved plan file (-out=FILE) or generated a fresh plan at apply time.
  • Whether -target was used. HashiCorp documents -target as intended for “exceptional circumstances, such as recovering from mistakes or working around Terraform limitations” and “not recommended … for routine operations, since this can lead to undetected configuration drift and confusion about how the true state of resources relates to configuration.” If the change used -target, that belongs in the record.

This section is where a lean team gets the most leverage. The typo is one character. The plan/apply gap is a process condition the team can change.

State assumptions belong in the postmortem

Terraform’s state documentation explains why state matters: “Terraform uses your workspace’s state to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. Terraform uses state to determine which changes to make to your infrastructure.” The same page recommends storing state in a remote backend with locking and access control, and warns against storing it in version control because that “can result in data loss or exposure of secrets stored in the state file.”

For the postmortem, the relevant questions are narrower:

  • Was the state current at plan time? If a manual change had occurred outside Terraform, was it reconciled before the apply?
  • Was state locking in effect, and did anyone else hold the lock during the apply?
  • Did the change depend on a data source or module output that could have shifted between plan and apply?

These are conditions, not accusations. A team that records “state was stale because a manual security-group edit was never reconciled” has found something it can fix — a refresh step in the runbook, a drift check, a policy about manual changes during incidents. A team that records “engineer made a typo” has found nothing it can fix except the engineer, and Google’s guidance is explicit that “you can’t ‘fix’ people, but you can fix systems and processes.”

Capture the detection and response timeline with exact times

PagerDuty’s postmortem template asks for specific timestamps: “Time the impact began,” “Time of the page,” “Time that the status page was updated,” “Time of any significant actions,” and “Time the impact ended.” Google’s example postmortem uses a screenplay-style timeline with entries at one- or two-minute intervals.

For a lean team, the most useful derived number is the detection lag: the interval between the change landing and the first signal. Record whether that signal was a synthetic check, an alert, or a customer report. The action item that follows is often about detection, not about the typo.

Also record the response path: who was paged, how long until acknowledgment, what the first diagnostic step was, and whether the rollback or reconcile path was documented. If the team has no dedicated reliability coverage, note that as a condition — the absence of a second pair of eyes on the apply, the absence of an automated drift check — rather than as an excuse. The postmortem’s job is to make the condition visible so it can be scheduled.

Give blast radius and recovery their own section

PagerDuty’s template asks for exact numbers under Impact: “Time in SEV-1,” “Time in SEV-2,” “Events Dropped / Not Accepted,” “Accounts Affected,” “Users Affected.” Google’s workbook chapter on postmortem culture criticizes a bad example for providing only duration and no size or scope, noting that “a well-informed estimate is better than no data at all.”

For a Terraform-triggered incident, blast radius has a specific shape: which resources were created, changed, or destroyed; which services depended on them; and whether the change touched state that other workspaces or manual processes also manage. Record the recovery path in the same section: was mitigation a revert, a re-apply, a manual console change, a state edit, or a restore from backup? If recovery depended on a manual step that had not been rehearsed, that is a finding about operational readiness.

If the incident involved data loss or a database, the recovery section should name the backup and restore mechanism and whether it was tested. PostgreSQL’s documentation on continuous archiving and point-in-time recovery describes the mechanics — WAL archiving, base backups, replay to a target time — but the postmortem’s job is not to restate the manual. It is to record whether the team’s actual restore path was known, owned, and exercised. A restore lottery that has never been run is a condition worth writing down.

Write action items across categories, with owners and finish lines

Google’s USENIX article on postmortem action items classifies them by category — “Investigate, Mitigate, Repair, Detect, Prevent” — “to make sure that the action item plan covers both very short-term and longer-term fixes.” The same article requires that “every postmortem stemming from a user-visible event” have “at least one P0 or P1 action item,” and quotes Ben Treynor Sloss: “To our users, a postmortem without subsequent action is indistinguishable from no postmortem.”

PagerDuty’s template adds the wording discipline: “Write all action items to be actionable, specific, and bounded. Create tickets for all action items and label with their severity level and date tags.” Google’s article gives the contrast directly: “Investigate monitoring for this scenario” is poorly worded; “Add alerting for all cases where this service returns >1% errors” is better. “Fix the issue that caused the outage” is poorly worded; “Handle invalid postal code in user address form input safely” is better.

For a typo-triggered incident, a balanced plan might include:

  • Detect: Add a synthetic check or alert that fires within N minutes when the affected resource becomes unreachable. Name the check and the threshold.
  • Prevent: Add a required plan-diff review step to the change checklist, or a CI job that posts the plan output to the pull request. Name the file and the reviewer role.
  • Mitigate: Document the rollback or reconcile procedure for this resource class in the runbook, with the exact commands and the expected output.
  • Repair: Reconcile the state drift that the incident exposed, and add a drift check to the weekly maintenance routine.
  • Investigate: Determine whether other workspaces share the same pattern, and record the result.

Each item needs a single owner, a ticket, a priority, and a finish line. “Be more careful with Terraform” is not an action item. “Add a plan-diff comment to the pull request template by [date], owned by [role]” is.

Keep the language blameless and the data specific

Google’s workbook chapter catalogs the failure modes of a bad postmortem: missing context, omitted key details, empty recovery sections, action items without owners, finger pointing, animated language, and delayed publication. Its examples of animated language — “careless ignorance,” “which is ridiculous,” “I can’t believe we survived this one!!!” — are worth keeping in mind on a small team where everyone knows who applied the change.

Blamelessness here is a writing discipline, not a disclaimer. Remove individual names from causal sentences. Replace adjectives with measurements. Keep the timeline factual. If the postmortem needs to say that a review step was skipped, say which step and what the checklist required, not who skipped it.

Publish quickly. PagerDuty’s template recommends scheduling the postmortem meeting “within 5 business days of any event.” Google’s workbook criticizes a postmortem published four months after the incident, noting that in the interim key details are forgotten. A short, honest postmortem published within days beats a comprehensive one published months later. For a lean team, the review meeting and the action-item tickets are the durable artifacts.

A hypothetical, clearly labeled

Suppose a team applies a one-line change to a security-group rule. The pull request showed a speculative plan with one rule change. At apply time, the engineer ran terraform apply without a saved plan, and Terraform generated a fresh plan. Because a manual console change from the previous week had never been reconciled, the fresh plan included an additional replacement of a database subnet group. The apply succeeded. The first signal was a customer report ninety minutes later.

The useful postmortem findings are not the typo. They are: the unreconciled state, the absence of a saved-plan step in the runbook, the missing drift check, and the ninety-minute detection lag. The action items follow from those findings. This example is constructed to illustrate the structure; it is not a record of a specific incident.

A short checklist for the typo-triggered postmortem

  • Separate trigger from contributing causes in the document structure.
  • Paste or link the reviewed plan and the applied plan; note whether they matched.
  • Record state assumptions: currency, locking, drift, data sources.
  • Record exact times for impact start, first signal, page, mitigation, and impact end.
  • Record blast radius with numbers or a well-informed estimate.
  • Record the recovery path and whether it was rehearsed.
  • Write action items across investigate, mitigate, repair, detect, and prevent; give each a single owner, a ticket, a priority, and a finish line.
  • Remove individual names from causal sentences and replace animated language with measurements.
  • Publish within days, not months.

What to leave out when evidence is thin: any claim about how common this class of typo is, any statistic about detection latency or action-item closure in small teams, and any assertion that a specific review tool would have prevented the incident. The sources describe general practice; they do not establish causal effects for this scenario. Record what happened, label what is inferred, and let the action items carry the weight.

For the recovery side of the same discipline — writing the checklist before the incident, so the postmortem can record whether it was followed — see Write the Recovery Checklist Before You Need It.

Sources