A runbook can be correct when written, still linked from the alert that invokes it, and wrong in the one step that matters. For a team of two to fifteen engineers who are their own on-call rotation, that is the expensive version of staleness: the document looks authoritative, the alert keeps firing, and the person reading it at 3 a.m. has no way to know which paragraph has quietly stopped matching production.
This is not a documentation problem in the abstract. It is a detection problem. Staleness is a property of the relationship between a runbook and the system it describes, not of the runbook’s age. A six-month-old runbook for a service that has not changed may be perfectly fresh. A two-week-old runbook for a service that shipped a new deploy path may already be wrong. The goal is to make that decay surface as a routine chore on a Tuesday afternoon rather than as a surprise during a customer-visible outage.
Why staleness is invisible by default
Runbooks are read under pressure, not on a schedule. The alert that points to one keeps firing, which creates a false sense of validation: the runbook is still reachable, so it must still be right. Nothing in the normal workflow compares the document to the system it describes. The person who wrote it may be the only person who has ever executed it, and their memory silently fills the gaps a fresh reader would hit immediately.
Google’s SRE Workbook is direct about the underlying rate of change. In its chapter on on-call, it states that playbook details go out of date at the same rate as the production environment changes, and that for daily releases a playbook might need an update on any given day. The same chapter notes that in SRE, whenever an alert is created, a corresponding playbook entry is usually created, and that these guides reduce stress, mean time to repair, and the risk of human error. That is the case for having runbooks. It is also the case for not trusting them indefinitely.
The workbook offers two broad strategies for keeping playbooks maintainable. One is to keep entries general enough that they change slowly — for example, a single entry for a class of alerts that a trained on-call engineer reads alongside an architecture diagram. The other is to write step-by-step procedures that reduce human variability and drive down MTTR. Both are legitimate. The workbook’s practical advice is that if a team agrees on nothing else, it should decide what minimal, structured details its playbooks must have, and notice when playbooks have accumulated a lot of information beyond those structured details. That minimal structured set is also the natural place to hang freshness signals.
Borrow the postmortem discipline
Google’s postmortem chapter contains a line that applies almost verbatim to runbooks: an unreviewed postmortem might as well never have existed. The chapter encourages regular review sessions so that each completed draft is actually reviewed, and it stresses that postmortem criteria should be defined before an incident occurs so everyone knows when a postmortem is necessary. It also lists a monitoring failure — which usually implies manual incident discovery — among the common triggers.
The same logic transfers. Define the runbook review trigger before you need the runbook. If the only time a runbook gets read is during an incident, then the incident is your review process, and you are paying for it in customer impact. A scheduled freshness check is cheaper.
Staleness signals worth checking
The following checklist is our own operational reasoning, not a measured result. It is offered as a template. The signals are chosen because each one can be checked mechanically or by a reader who did not write the runbook.
- Last-verified date and named owner. A runbook without an owner has no one to notice when it drifts. A date without a name is a timestamp, not accountability. A vendor guide on incident response runbooks recommends scheduling quarterly reviews, making version control standard, and keeping ownership visible so the runbook reflects the current state of the infrastructure. The cadence is a recommendation, not a finding; the ownership point is the durable part.
- The alert or dashboard the runbook is bound to still exists and still fires. If the alert was renamed, retired, or split, the runbook is orphaned even though its text is unchanged. Orphan detection is a mechanical check and can be automated with a link and asset linter.
- Every command still parses and its expected output still matches. A command that returns a different shape of output than the runbook says to expect is a stale runbook, even if the command itself succeeds.
- Every linked dashboard, log query, and escalation path still resolves. Dead links are the cheapest staleness signal to detect and the easiest to ignore.
- The rollback step still corresponds to the current deploy mechanism. Deploy paths change more often than most other parts of a service, and the rollback step is the one you least want to discover is wrong.
- The stated blast radius still matches the service’s current dependencies. A runbook that says “affects the checkout service” when the service now also fronts authentication is understating the incident.
One more signal is worth treating as a smell in its own right: if the runbook is a fixed list of commands run every time a particular alert fires, that is a sign the runbook is standing in for automation. The SRE Workbook makes this explicit — if playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, it recommends implementing automation. A runbook that should be a script will drift every time the underlying command surface changes, and it will drift silently.
A cheap freshness drill for a small team
Pick one runbook per week. Have someone who did not write it execute it against a non-production or read-only path. Record what broke. That is the whole drill.
This is a rehearsal of the document, not of the recovery. Keep it separate from your backup and recovery drills so neither dilutes the other. A restore lottery tests whether your backups can be restored; a runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways and should not share a slot.
The cadence should be sized to the team, not copied from a large-org quarterly cycle. One runbook per week per rotation is a workable default for a small team. The point is that the cadence exists and is owned, not that it matches a specific number. If your team is already running a recovery checklist, the same weekly slot can host both, as long as the two exercises stay distinct. The Recovery Checklist article is a useful companion for the restore side; the freshness drill here is the documentation side.
Keep the freshness check read-only or non-production wherever possible. The goal is to detect drift, not to add a second, riskier change path to production.
What to do when a runbook fails its check
Treat it as an owned follow-up with a due date, not a documentation chore. The postmortem discipline of assigning follow-up actions applies equally to runbook maintenance. The difference is that runbook decay is predictable and can be scheduled rather than discovered. A vendor guide on incident response runbooks puts it plainly: treat every incident as a freshness test, and let stale instructions create an owned follow-up. That is the right instinct, but it is reactive. The scheduled drill is what makes it proactive.
If the runbook repeatedly fails its freshness check, that is information. Three options are available: rewrite it so the steps are stable, retire it and replace it with a pointer to the automation that now handles the case, or automate the deterministic parts and keep the runbook for the judgment calls. The SRE Workbook’s recommendation to automate deterministic command lists is the strongest signal that a runbook has outlived its form.
There is a related trap worth naming. PostgreSQL’s documentation notes that WAL archiving restores changes to the database but not changes to configuration files such as postgresql.conf, pg_hba.conf, and pg_ident.conf, since those are edited manually rather than through SQL operations. A runbook that says “restore from WAL” without saying where the configuration files come from is stale in a way that will only show up during a real recovery. The same documentation advises monitoring the archiving process to ensure it is working as intended, and warns that if archiving falls significantly behind, the amount of data that would be lost in a disaster increases. That is a monitoring signal, but it is also a runbook signal: if the runbook does not tell the responder how to check archive lag, the runbook is incomplete.
Where the published guidance already agrees
The SRE Workbook’s playbook section is the closest primary source for this argument. It acknowledges that playbook detail decays at the rate of production change, that teams should agree on minimal structured fields, and that deterministic command lists should be automated. The postmortem chapter supplies the review discipline: unreviewed artifacts might as well not exist, and criteria should be defined before the incident. A vendor guide on incident response runbooks adds the operational habits — scheduled reviews, version control, visible ownership, and treating incidents as freshness tests — though its comparative tool claims are marketing material and should not be read as independent evidence.
What none of these sources provides is a measured rate of runbook staleness in the field, or a measured effect of scheduled freshness checks on incident outcomes for teams of this size. That evidence does not appear to exist in the retrieved material. The checklist above is therefore a proposed practice, not a validated one. It is cheap enough to try, and the failure mode it addresses — a runbook that is wrong in the one step that matters — is expensive enough to justify the experiment.
FAQ
How often should a small team review runbooks?
There is no evidence-based cadence for a 2–15 engineer team. One runbook per week per rotation is a workable default because it distributes the load and keeps the review short. The important property is that the cadence is owned and visible, not that it matches a particular interval. A quarterly cycle copied from a larger organization will likely be too infrequent for a service that deploys daily.
Should the freshness check be part of the on-call shift?
It can be, but it should not be the on-call engineer’s only documentation task. The value of having someone who did not write the runbook execute it is that they hit the gaps the author’s memory fills. If the same person writes and checks every runbook, the check degrades into a reread.
What is the difference between a runbook freshness check and a restore drill?
A restore drill tests whether your backups can be restored. A runbook freshness check tests whether the instructions a responder would follow still describe the system. They fail in different ways. A restore drill can pass while the runbook that tells you how to run it is wrong, and a runbook can be accurate while the backup it references is unusable. Keep them separate.
When should a runbook be retired rather than updated?
When the procedure it describes has been automated, or when the service it describes no longer exists. The SRE Workbook’s guidance to automate deterministic command lists is a useful trigger: if the runbook is a fixed list of commands run every time an alert fires, the runbook is a candidate for replacement by a script. Retiring it is not a loss; it is the intended outcome.
Does a stale runbook actually cause longer incidents?
The retrieved sources do not measure this. The SRE Workbook states that playbooks reduce stress, MTTR, and the risk of human error, which implies that a wrong playbook does not. But the specific claim that stale runbooks lengthen time-to-mitigation is not established by the evidence reviewed here. Treat it as a reasonable hypothesis, not a finding.
The honest limit
None of this prevents the incident where the runbook was fine and the system was not. The goal is narrower: make sure that when a runbook is wrong, you find out on a Tuesday afternoon and not during a customer-visible outage. For a lean team with no dedicated reliability coverage, that is a worthwhile trade.