by ·Comments Off on Thegrayhaven — Your Community Hub

Thegrayhaven — Your Community Hub

Local news, events, and resources for an engaged community.

Browse Latest Posts

Strong communities don’t happen by accident. They grow when neighbors actually know each other’s names. When people show up to town halls instead of just complaining online later. This site exists to bridge that gap — connecting residents, highlighting local opportunities, and making civic life less of a mystery. We cover local news that actually matters to your daily life, maintain event listings so you don’t miss the good stuff, and try to answer those “who do I call about…” questions that come up way too often.

Topics we cover: Local News · Events · Resources · Projects · Directory · Contact

by ·Comments Off on Code Fix or Workaround: How to Decide When a Recurring Failure Deserves the Real Repair

…

…”
}

Let me write it all out now, carefully.

Also, minor: “2 a.m.” — keep. “Backblaze B2” keep. “pgBackRest 2.50” keep. “restic 0.16.4” keep. All factual claims preserved.

One small humanization: “We have been burned both ways.” — I changed to “we have been burned both ways” — keep contraction “we’ve been burned both ways”? Contraction reads more natural: “and we’ve been burned both ways.” Good.

Also in the intro, “hands you the same fork” — good.

I think the piece is ready. Write final JSON, single html string.{
“title”: “Code Fix or Workaround: How to Decide When a Recurring Failure Deserves the Real Repair”,
“html”: “

Every lean team collects failures that refuse to stay dead. A nightly restic job dies on a stale lock. A report endpoint exhausts its connection pool every second Friday. A managed queue throttles without so much as a warning. Each recurrence hands you the same fork: repair the cause in code and configuration, or absorb it with an operational workaround — a scripted step, a runbook line, an alert threshold that keeps the symptom at arm’s length. We call this the fix-or-workaround decision, and on a team of two to fifteen engineers with no dedicated SRE function, nobody owns it by default. It gets made by whoever holds the pager at 2 a.m. Tired, and reaching for the fastest exit. That default is expensive in both directions, and we’ve been burned both ways. The wrong workaround becomes invisible toil that outlives its author. The wrong fix burns scarce engineering time on a failure mode that never earned it. This field note is the framework we use instead — five evidence signals, two anonymized incidents, and a one-page decision record that keeps the choice from quietly expiring.

A small engineering team reviewing incident data together at a laptop during an operations review
The decision gets made with evidence on the table, not by whoever is holding the pager.

What counts as a recurring failure — and what does not

A recurring failure is one failure mode observed three or more times — not three failures that merely look alike. The distinction matters, because the fix-or-workaround decision only pays off when you’re confident the occurrences share a mechanism. Our test is mechanical. Every incident note gets a failure-mode tag — backup-lock-stall, pool-exhaustion-report — and the tag only goes on if the responding engineer can name the mechanism, not just the symptom. Three separate 5xx spikes with three different causes are three first-time incidents. Three spikes where the same report endpoint holds PostgreSQL connections open past its statement timeout are one recurring failure with a growing bill.

An operational workaround lives outside your codebase: a runbook step, a wrapper script, a scheduled job, a raised threshold, a bigger instance. It compensates for the failure mode without removing it. A code fix removes the failure mode at its source — application code, schema, infrastructure configuration, or a dependency version bump. The tradeoff never changes, so we’ll state it plainly. Workarounds are cheap to install and cheap to reverse, but they charge operator attention on every recurrence. Fixes are expensive to install, carry rollout risk, and are usually permanent. Neither choice is virtuous by default. A team that only ever works around accumulates toil it can’t see. A team that only ever fixes spends its scarcest budget on failure modes that may not deserve it.

The five signals we score before deciding

When a tagged failure mode hits its third occurrence, we spend about thirty minutes scoring five signals. The score doesn’t decide for you. What it does is put the tradeoffs on the table before someone burns a weekend writing a fix nobody asked for — or before a workaround quietly becomes one person’s muscle memory and nobody else’s.

1. Recurrence cost, in operator minutes

Multiply frequency by handling time over a rolling month. A failure that recurs twice monthly and takes twenty minutes of on-call effort to clear costs about forty operator minutes a month. If a credible fix runs three engineer-days — roughly 1,440 minutes — the fix pays for itself in handling time alone after about three years. Weak case, unless another signal intervenes. Our working thresholds, tuned on small teams: below roughly thirty operator minutes a month, a workaround is almost always right; above two hours a month, the fix gets scheduled regardless of anything else; between those two, blast radius decides. Treat the arithmetic as a floor on the fix’s value, not a ceiling. It deliberately undercounts interrupted sleep, context switching, and the drift that sets in when the same person is always the one who knows the incantation.

2. Blast radius: does it touch the data path?

Failures that only degrade convenience tolerate workarounds well. Failures that can corrupt, truncate, or lose data do not. Our hard rule: any recurrence with a plausible data-integrity path — a WAL archive gap, a backup that fails verification, a write path that can partially commit — earns either an immediate fix or a workaround that is itself alarmed. A compensating control that can fail silently isn’t a control. PostgreSQL point-in-time recovery depends on an unbroken WAL chain, whether that chain is managed by pgBackRest 2.50 or by native continuous archiving, as the PostgreSQL documentation states plainly. An archiving stall that a cron job clears by restarting the archiver is acceptable only if a second, independent check confirms the chain actually resumed. A silently gapped archive is exactly the failure class a restore lottery drill exists to catch — before a real restore does.

3. Where the ownership boundary sits

If the defect lives in a dependency you don’t maintain, the code fix isn’t yours to write. The decision narrows to four levers: pin a version, change a configuration, open an upstream issue with your evidence attached, or wrap the boundary with a script. The stale-lock behavior in restic belongs to restic. A team running restic 0.16.4 manages that behavior; it does not repair it. When the failure mode crosses into code you own, the real fix is on the table, and the cost estimate in signal five begins.

4. Whether the workaround can be observed

A workaround nobody can see becomes invisible toil. It runs in one person’s memory, dies with an offboarding, and resurfaces months later as a mystery outage. Every workaround we register gets at least one of three signals — a synthetic check, a metric, or a low-urgency PagerDuty alert that fires when the workaround is invoked. If a candidate workaround can’t be made to emit any trace, that isn’t grounds for instant rejection, but it is evidence for the fix. Google’s SRE workbook recommends keeping toil below half of an engineer’s time before it crowds out project work, and you can’t manage that budget against workarounds you can’t count.

5. What the fix actually costs

Estimate the fix as writing, review, deployment, a rollback plan, and verification — not as the diff. On a four-person team, a two-line change to a billing-critical path is not a two-line change. It’s a release with money on the line and no second reviewer awake. Ask what the fix can break, too. A pool ceiling that stops exhaustion can convert hard 500 errors into slow, soft queuing that nobody notices for a week. Fixes have blast radii, and an estimate that ignores them is marketing, not arithmetic.

Two engineers comparing notes beside a laptop while working through a fix decision
Two incidents from our field notes, two different verdicts.

Two incidents, two different answers

Same fork, opposite verdicts. Here is what tipped each call.

The stale restic lock: the workaround won

A six-person B2B scheduling team we advised runs nightly restic 0.16.4 backups to Backblaze B2 over a WAN link that drops a few times a month. An interrupted run occasionally left a lock behind, and the next night’s job failed with repository is already locked — an error that looks like a security incident at 2 a.m. and almost never is one. Recurrence: roughly every six weeks, about twenty-five minutes of on-call handling, no exposure to the data path. And the fix did not exist to be written. Lock semantics belong upstream, so signal three removed the code fix from the menu entirely.

They wrote a forty-line wrapper instead. Verify no live restic process, run restic unlock, retry the backup, emit a metric, and open a low-urgency PagerDuty alert if two consecutive nights fail. Recurrence cost dropped from twenty-five operator minutes per event to zero pages and about one ticket a quarter. The decision record reads: workaround is the permanent answer; revisit at the next restic minor release; reopen if the alert fires more than once a quarter, because rising frequency is the tell that this arrangement is rotting.

The Friday pool exhaustion: the fix won

A four-person fintech team saw 5xx bursts roughly every nine days, cleared by an app-tier restart in about fifteen minutes. The arithmetic said workaround: about sixty operator minutes a month against a fix estimated at two engineer-days, and the mean time to restore looked respectable on the monthly report. That’s part of why the restart step survived a full quarter. Two signals overrode the arithmetic. The blast radius touched the data path — a report query held connections open long enough to starve transactional traffic — and the workaround was masking a trend: the interval had shortened from fourteen days to nine inside a single quarter. A recurring failure whose interval is shrinking is a fix candidate regardless of what the cost model says, because the model prices today’s frequency, not next quarter’s.

The fix was unglamorous. A statement timeout on the report path, a pool ceiling, and a Grafana panel on pool saturation so the trend stayed visible after the symptom disappeared. Two engineer-days, one rollback plan, and a canary deploy on a Friday morning rather than a Friday evening. The team also retired the restart runbook step that same week — a workaround and its fix shouldn’t coexist past the deploy that makes the workaround dead weight.

Write the decision down, or you never made it

A fix-or-workaround decision that lives in a Slack thread is a decision that expires silently. We keep a one-page decision record per recurring failure, stored beside the runbooks, with six fields: date, failure-mode tag, evidence (occurrence count, operator minutes, trend), the decision itself, the expiry trigger that will reopen the question, and an owner. It pairs naturally with the habit of writing the recovery checklist before you need it — every registered workaround is a line item on that checklist, and a checklist step with no owner and no alert is the same silent toil in a different binder.

When the decision lands on the workaround side, run a fifteen-minute pre-mortem before closing the record. The prompt is one sentence: it is eight months from now, and this workaround just made an incident worse — what happened? The answers tend to be specific. The wrapper script never made it into the new-hire runbook. The low-urgency alert got muted during a release and never unmuted. The unlock step ran against the wrong repository after a storage migration. Each answer becomes a guardrail line in the record.

Expiry triggers we use — any one of these reopens the decision:

  • Recurrence crosses the two-hour monthly operator threshold.
  • The workaround fires during a customer-visible incident. Even once.
  • A new hire can’t execute the runbook step in a single pass during onboarding.
  • Upstream ships a fix, or the pinned dependency approaches end of life.
  • The registry doubles in size within a quarter — a system-level sign the team is deferring rather than deciding.

Audit the registry quarterly. The count of active workarounds is a health metric, not a trophy. A small number with alerts and owners is normal operations. A growing number with neither is debt compounding at on-call interest rates.

An engineer writing in a notebook next to a laptop during a runbook review
Every registered workaround carries an owner, an alert, and an expiry date.

When the workaround is the correct final answer

Some failure modes should never get a code fix, and saying so out loud keeps the registry honest. Four cases recur in our notes. First, upstream-owned defects, where your whole lever is a version pin and an issue number. Second, systems scheduled for replacement: fixing a report path you plan to delete next quarter is investment in the wrong asset, and a documented workaround with an expiry date is the disciplined answer. Third, low-frequency failures with a high fix cost where the blast radius is fully covered by an alarmed control — a synthetic check that catches the degradation in under a minute can make a quarterly manual step acceptable. Fourth, vendor-managed limits and quotas, where the only lever you actually own is the alert that tells you before the customer does.

The cautious framing matters more than the list, though. Workarounds compound. Each one costs a little attention forever, and attention is the one budget a team of five can’t print. A workaround that was right in March can be wrong by October because headcount, traffic, or the dependency changed underneath it. That’s why the record carries an expiry trigger and the registry carries an audit date. The decision is renewable, not permanent, and renewing it costs thirty minutes.

The habit, compressed

Tag the failure mode. At the third occurrence, score the five signals: operator minutes, blast radius, ownership, observability, and true fix cost. Decide. Record the decision with an owner and an expiry trigger, and pre-mortem any workaround you keep. The whole ceremony fits inside a lunch break, and it replaces the two expensive defaults — the 2 a.m. reflex and the weekend fix nobody scoped.

Frequently asked questions

How many recurrences justify a code fix?

Three verified occurrences is the entry bar for making the decision, not a mandate to fix. A recurrence with a plausible data-integrity path can justify a fix on the first occurrence; a cosmetic twenty-minute failure can recur monthly for years and still lose to a well-alarmed workaround. The count gets you into the room. Operator minutes and blast radius hand down the verdict.

How do you keep workarounds from becoming invisible toil?

Register each one, make its invocation emit a signal — a metric, a synthetic check, or a low-urgency PagerDuty alert — and audit the registry quarterly with expiry triggers attached. A workaround that leaves no trace when it runs is the definition of invisible toil, and the inability to instrument it is itself evidence for writing the fix.

What if the fix belongs to a dependency you do not maintain?

Then the decision is about the boundary, not the interior. Pin the version, change the configuration, open an upstream issue with your occurrence data attached, and wrap the boundary with a script that absorbs the symptom. In the restic example, the wrapper script is the entire fix available to that team, and recording it as a deliberate boundary decision beats pretending a deeper repair is on the roadmap.

Does every recurring failure need a full postmortem?

No. The first occurrence gets an incident note; the second gets the failure-mode tag and a five-minute check that the first note named the right mechanism; the third gets a full blameless postmortem with action items tracked to closure. Google’s SRE book makes the case that postmortems only pay off when action items are followed through, and a recurring failure is exactly the case they were built for. The action item list from that third postmortem is your fix-or-workaround queue, pre-sorted by evidence.

Got a recurring failure sitting on the fence — one where the arithmetic says workaround and your gut says fix? Send us the details, anonymized. Scoring reader-submitted calls is becoming a regular column here, because the fastest way to calibrate a framework is to run it against failures we didn’t choose ourselves.

by ·Comments Off on Code Fix or Operational Workaround? A Decision Procedure for Recurring Failures

Every recurring failure eventually forces the same decision: change the code, or change how you operate around it. A recurring failure is a failure mode that has returned after you believed it was resolved — the third 3 a.m. page for the same PostgreSQL connection exhaustion, the second month with WAL archiving stalls, the weekly restic job that dies on a stale lock. The fix-or-workaround decision is the judgment call about whether the next occurrence gets a merged pull request or a runbook entry, and it touches everything else a small team manages: MTBF and MTTR tracking, error budgets, workaround debt, blameless postmortems, and runbook maintenance. For a team of two to fifteen engineers running production without a dedicated SRE function, this decision arrives at least monthly, usually under time pressure and from memory. This article replaces memory with a procedure: recurrence thresholds, a cost line you can compute in ten minutes, three anonymized incidents with three different verdicts, and a register format that keeps workarounds from rotting.

The short answer

A recurring failure deserves a code fix when three conditions hold at once. It has recurred at least three times in ninety days, or it has breached a stated guarantee such as your RPO. The fix has a bounded scope you can estimate in engineer-hours. And the recurring cost of living with the failure — pages, manual recovery, workaround maintenance — exceeds the one-time cost of the fix within roughly two quarters. It deserves an operational workaround when any one of those conditions fails: the root cause sits in code you do not own, the fix is unbounded, or the workaround is cheap, durable, and reversible.

Both paths are legitimate. The failure mode to avoid is the unmade decision — the workaround nobody agreed to, written nowhere, reviewed never, quietly becoming permanent infrastructure. Everything below is about gathering evidence for those three conditions and keeping the workaround path honest.

Count recurrences before you debate solutions

Arguments about fixes are usually arguments about frequency conducted from memory. Replace the memory with a count. Export the last ninety days of incidents from PagerDuty or your paging tool, tag everything belonging to the same failure mode — one tag such as db-conn-exhaust is enough — and count. If you have never tagged incidents, start with the failure you already suspect; you will likely find more occurrences than anyone remembered.

Weight the count by impact. Three warnings logged during business hours are not equivalent to three pages at 3 a.m. Google’s SRE book treats unreliability as an error budget — a fixed quantity you may spend per quarter — and the metaphor works even without formal SLOs: each recurrence spends attention you do not get back, and a tag that has paged the on-call three times in a quarter has spent enough.

Record MTBF (mean time between failures) and MTTR (mean time to restore) from the incident timestamps. One reader team of four tracked a nightly restic lock failure for six weeks before deciding anything: MTBF 1.6 days, MTTR eleven minutes, all during business hours, zero user impact. That profile argues for patience. The same failure with a 2 a.m. median page time flips the arithmetic entirely.

The cost line that settles most debates

Compute a monthly cost in plain units: recurrences per month × (MTTR in minutes × loaded engineer rate per minute) + user impact. At a $90/hour loaded rate, a minute costs $1.50. A failure paging four times a month at twenty-five minutes per recovery costs $150 in labor alone; add the user-facing minutes and the total usually lands between $200 and $400. Compare that against the fix estimate multiplied by 1.5 — the multiplier is the honest buffer for review, testing, and the regression you will cause. A sixteen-hour fix at $1,440 breaks even against a $300 monthly cost inside five months. Break-even beyond a year means workaround with a review date, not fix. The point is not precision; it is that the debate moves from adjectives to two numbers anyone can check.

Two engineers at a workstation reviewing incident graphs together

Signals that point to a code fix

  • It violates a guarantee you have written down. Backups restorable within your stated RPO, WAL archiving with no gap past your threshold, checkout completing in under two seconds. Guarantees outrank cost arithmetic; a workaround that masks a violated guarantee is an outage you have scheduled.
  • The workaround requires a human to remember something. Runbooks reduce that cost but do not remove it; memory is not a control.
  • The fix touches code you own and has reviewable scope. A diff under a few hundred lines that one engineer can write, review, and test inside a week qualifies.
  • The failure delays or corrupts data. Anything discovered late — a two-day-stale backup found during a restore drill — counts double.
  • You have patched the same seam twice and it regressed. Repeated patches at the same spot mean the design is wrong, not the patch.

Signals that point to an operational workaround

  • The root cause is in a dependency you do not control. A vendor API returning 429s at month-end, intermittent mount behavior from a cloud provider, a flaky upstream resolver. You cannot patch code you do not own; you can absorb its behavior.
  • The workaround is reversible configuration. PgBouncer pool sizing, a systemd timer restart, retry-with-backoff at the proxy, an archive_timeout value. Config with rollback is cheap to test and cheap to undo.
  • The failure is rare relative to the fix. Once a quarter with a five-minute recovery does not justify a migration that consumes a month of one engineer’s capacity.
  • The code fix requires a migration. Schema changes, queue rewrites, and storage swaps carry risk a small team absorbs poorly; the workaround buys time to plan them properly.

A small team at a conference table working through a postmortem document

Three incidents, three verdicts

The archive gap that was not a code bug (PostgreSQL 16, pgBackRest 2.50)

A six-person team on GCP was flagged by a synthetic check for WAL archive gaps longer than fifteen minutes, two to three times a month. The instinct was a pgBackRest bug: the check measured the distance between the current WAL position and the last archived segment, and the gaps looked like data at risk. The evidence disagreed. Gaps clustered between 2 a.m. and 4 a.m., when write volume fell and segments were not filling fast enough to switch. The fix was operational in two parts: archive_timeout = 300 to force a switch every five minutes, and a corrected check threshold that told a slow segment apart from a stalled archiver. No application or backup code changed. Verdict: fix the monitoring and the configuration, not the software.

The vendor API nobody could patch (retry queue)

A three-person team integrating a tax calculation API saw 429 and 503 bursts during the vendor’s month-end peak. The root cause was demonstrably not theirs, the vendor would not commit to a fix date, and the failure was absorbable: requests were not time-critical inside a two-hour window. They wrote a forty-line worker that queued failed requests and retried them with exponential backoff, added a queue-depth alert, and documented the flow in the runbook. Verdict: a permanent operational workaround, reviewed quarterly, owned by name. This is what a good workaround looks like — an owner, an alert, and a review date.

The workaround that masked a broken guarantee (restic 0.16)

A two-node bare-metal team ran nightly restic backups on a systemd timer. Interrupted runs left stale locks, and the on-call developed a habit: run restic unlock by hand, acknowledge the page, move on. This went on for six weeks. Their monthly restore lottery — a drill that picks a random snapshot to restore and verify — then produced a snapshot two days older than expected, because the previous night’s run had failed on a lock and the alert had been acknowledged rather than investigated. The cost line had crossed weeks earlier; the drill made it visible. The fix was twenty lines: a pre-run lock check that unlocked only when no restic process was active, enforced with systemd unit ordering. Merged in an afternoon, reviewed by the second engineer, recurrence stopped. Verdict: code fix, overdue.

The pattern across the three: the verdict rarely matches the instinct. The suspected software bug was a threshold problem. The obvious code fix was impossible. The harmless manual habit was eroding the one guarantee the team had written down.

Workarounds need expiry dates

An undocumented workaround is a future incident with a delay timer. Treat every workaround as a decision with a review attached. The lightest format that works is a single page in the ops repository — a register with one row per workaround: the failure mode it absorbs, the named owner, the monitoring signal that detects when the workaround itself breaks, and a review date. A three-paragraph decision record next to each row keeps the reasoning findable: context (the recurrence count and cost line), decision (workaround, plus the conditions that would reverse it), and consequences (what you accept until the review date).

Two rules keep the register honest. First, no workaround ships without a runbook entry and a monitoring signal — a workaround you cannot detect failing is a blind spot, not a mitigation. Second, anything older than two quarters gets a re-decision rather than renewal by silence. The runbook entry deserves the same care as a recovery checklist written before it is needed; the discipline transfers directly.

Who makes the call, and when

Not during the incident. During an incident the on-call restores service with whatever is fastest and safest, and a workaround applied under pressure is legitimate. The fix-or-workaround decision belongs to the postmortem’s action-item review, within a week, with recurrence data on the table. Google’s SRE book argues that blameless postmortem culture produces better action items than accountability-driven reviews, and Etsy’s 2012 piece on blameless postmortems remains the clearest short statement of why: engineers who expect blame report less, and unreported failures compound.

In a team of two to fifteen, the decision meeting is thirty minutes with two people: the on-call who handled the last occurrence and the engineer who owns the service area. The output is the decision record, not a consensus memo. When those two disagree after looking at the same cost line, the tiebreaker is the guarantee question — does the failure violate something you have promised in writing, to yourself or a customer? Guarantees win.

The procedure, end to end

The whole method compresses into seven steps small enough to run during one ops review:

  1. Tag the failure mode and count recurrences from the last ninety days of paging history.
  2. Record MTBF and MTTR from incident timestamps; note time-of-day and user impact for each occurrence.
  3. Compute the monthly cost line: recurrences × (MTTR in minutes × loaded rate) + user impact.
  4. Estimate the code fix in engineer-hours, multiply by 1.5, and flag anything unbounded — migrations, schema changes, vendor coordination.
  5. Answer the two qualifying questions: do you own the code that produced the failure, and does the failure violate a stated guarantee?
  6. On the fix path, schedule the change inside two sprints and link the decision record to the postmortem.
  7. On the workaround path, write the runbook entry, add the monitoring signal, set the review date, and add the row to the register.

For a shorthand version, the signals compress into one table:

Signal Leans toward Why
Three or more pages in ninety days Code fix Cost compounds, and alert fatigue is real
Root cause in a vendor or dependency Workaround You cannot patch code you do not own
Workaround needs a human to remember a step Code fix Memory is not a control
Failure violates a written RPO or RTO Code fix Guarantees outrank estimates
Fix requires a multi-week migration Workaround with review date Capacity is finite; buy time deliberately
Workaround is reversible config Workaround Cheap to test, cheap to undo

An engineer at a laptop updating a runbook between alerts

Frequently asked questions

How many recurrences justify a code fix?

Three in ninety days is the working threshold for a small team, because three is enough to compute a meaningful MTBF and enough to prove the first fix did not hold. Impact modifies the count: one recurrence that breached your RPO during a restore drill outranks five benign warnings, and a failure mode that pages at night counts double. Treat the threshold as the entry point to the cost line, not a substitute for it.

What separates a workaround from a band-aid?

Evidence. A workaround is documented in the runbook, detected by a monitoring signal when it breaks, owned by a named person, and reviewed on a date. A band-aid is none of those things. The test: if a new on-call engineer hit the failure tonight, could they find the workaround in the runbook and know whether it is still sanctioned? If not, it is a band-aid, and it is accruing debt at interest.

How do you keep workarounds from becoming permanent?

Expiry dates and a monthly read-through. Every workaround gets a review date at creation — ninety days is a reasonable default — and the register is read aloud in the ops review. Anything past its date gets an explicit re-decision: fix it, extend it with a written reason, or accept it permanently and promote it into the runbook as normal operating procedure. Silence never renews a workaround.

Should the on-call engineer make this call during an incident?

No. During an incident the on-call restores service with whatever is fastest and safest, including a workaround, and that judgment is not second-guessed afterward. The fix-or-workaround decision happens in the postmortem review within a week, with the recurrence data in hand. Deciding mid-incident optimizes for adrenaline; deciding in the review optimizes for the pattern.

When is a workaround better than a code fix even though you could fix the code?

Three situations recur in practice: the root cause sits in a dependency whose behavior you can absorb but not change; the code fix requires a migration whose risk exceeds the failure’s cost; or the failure is rare — roughly once a quarter with a five-minute recovery — so the fix would be the most expensive line item of the quarter for that service. In each case the workaround is a deliberate, documented, reviewed decision rather than a deferral.

Where to start this week

Export the last ninety days of pages, tag the top three failure modes by count, and run the cost line on each. Take the one with the worst ratio of monthly cost to fix estimate and write its decision record — three paragraphs, thirty minutes. If the verdict is a workaround, the runbook entry and its monitoring signal come first; the template we use is the one in Write the Recovery Checklist Before You Need It. If the verdict is a fix, schedule it before the next occurrence argues with you.

Recurring failures are the most legible data a small team has about where its systems actually hurt. Deciding fix-or-workaround on a cadence, with two numbers and two qualifying questions, turns that data into the only kind of reliability progress a team without an SRE department can sustain: small, boring, and repeatable. If you run the cost line on a real failure and disagree with your own verdict, send us the anonymized numbers — working reader verdicts is the column we want to run next.

by ·Comments Off on Code Fix or Operational Workaround? A Decision Test for Recurring Failures

…

…

…”
}

I’ll join blocks with
(escaped as literal backslash-n in the JSON source). Writing it out.

Also note: within HTML I have curly braces in “any two of {three or more occurrences, …}” — fine in HTML/JSON.

Let me write the final JSON now. I’ll be meticulous with escaping.

Actually, one more consideration on “harden” the JSON: avoid stray newlines inside the JSON string. I’ll write the entire JSON as a single line? That could be very long but valid. Or multi-line JSON where the string is one line. I’ll write the JSON object across lines but keep the html string on one line (with
escapes inside). That’s valid JSON.

Let me compose.{“title”:”Code Fix or Operational Workaround? A Decision Test for Recurring Failures”,”html”:”

When the same failure shows up for the third time in a quarter, a team of five engineers faces a real decision: change the code, or write a runbook step and move on. The short answer we use at Gray Haven Lab: fix the code when the failure is frequent, customer-visible, or quietly eroding your detection. Keep the workaround when the failure is rare, cheap to absorb, or the fix would be larger than the failure it prevents. This article is the decision test we run — five questions, concrete thresholds, and the carry-cost math that keeps the debate under ten minutes.

Two definitions so the test stays unambiguous. A code fix is any reviewed, tested change: application code, a Terraform module, a Kubernetes manifest, a database migration. An operational workaround is a documented response that lives outside the deploy path: a runbook entry, a remediation script, an alert with attached steps, a manual procedure with a named owner. Neither is inherently better. The question is which one the failure pattern has earned.

Laptop screen showing source code in an editor

Why small teams default to the workaround — and why that is mostly fine

In a 2–15 person team, engineer-hours are the scarcest resource, and every code fix competes directly with the roadmap. A workaround costs minutes per occurrence; a fix costs hours or days up front. Defaulting to the workaround is rational capacity management, not laziness.

But workarounds carry costs that never show up in a sprint plan: attention fragmentation for whoever is on call, alert fatigue when the same page arrives with no resolution in sight, and tribal knowledge when the workaround lives in one person’s head instead of a runbook. The failure mode is not choosing a workaround — it is choosing one silently, without an owner, a counter, or a review date.

The five-question decision test

Run this in the incident review, not during the outage. It takes ten minutes with your incident history open in PagerDuty, incident.io, or a plain incidents.md file in the repo. We describe the full runbook-first incident response setup elsewhere; this test slots into the same review.

1. How often does it actually happen?

Count occurrences with the same failure signature — the same error text, the same component, the same recovery step. Fewer than two in the last 90 days is a workaround candidate. Three or more is a fix candidate. The exact threshold matters less than having one written down; ours is three, because twice can be coincidence and three times is a pattern you can graph.

2. What does each occurrence cost?

Multiply: minutes of toil per occurrence × people involved × occurrences per year. A workaround that takes 20 minutes of on-call attention twice a month costs roughly eight engineer-hours a year. A fix estimated at three engineer-days pays back in about four and a half years at that rate — usually not worth it. Flip the frequency to weekly and the same fix pays back in three months. This carry-cost comparison is the most useful number in the conversation, and it is the one teams skip most often.

3. Does it touch customers, money, or data?

Internal failures and customer-facing failures are not the same currency. If the failure consumes error budget against an SLO — checkout availability on a Grafana dashboard, for example — or risks SLA credits, one customer-visible occurrence outweighs ten internal ones and the bar for a fix drops sharply. A recurring deadlock in a nightly batch job is an annoyance; the same deadlock on the order path during business hours is a fix, full stop.

4. Is the fix bounded?

A retry with a regression test is a bounded fix. A change that requires a schema migration on a hot Postgres table, or a rewrite of the session layer, is not. Frequent failure plus bounded fix: fix it now. Frequent failure plus unbounded fix: workaround now, with a scheduled revisit and the design problem on the engineering backlog. Rare failure plus bounded fix: fix it opportunistically the next time the file is open anyway. Rare failure plus unbounded fix: workaround, and stop feeling bad about it.

5. Is the workaround hiding the failure?

The most expensive workaround is the one that works so well nobody records it anymore. If the response has become muscle memory — no incident entry, no tally, just a quick kubectl rollout restart — you have lost the evidence you would need to revisit the decision. Keep the alert. Keep the count. A workaround with a rising tally is a fix in disguise.

Scoring: any two of {three or more occurrences, customer-facing impact, bounded fix} point to a code fix. Otherwise, document the workaround with an owner and a review date. If the team is split, write down both carry-cost estimates and check the tally again after the next two occurrences — the data settles it faster than another meeting.

What a good workaround looks like

  • A runbook entry with the failure signature. Exact error text, the component, the commands, and one verification step. Keep it in a runbooks/ directory in the repo or in your docs tool; the location matters less than the link from the alert.
  • An alert that points to the runbook. Prometheus alert rules support a runbook_url annotation, and PagerDuty and Opsgenie both let you attach it to the escalation. The on-call engineer should never reconstruct the procedure from memory at 2 a.m.
  • An owner and a review date. Every workaround gets a named owner and a slot in the quarterly open-workarounds review. Fifteen minutes, the list on a board, promote or retire each item.
  • A counter. A tally line in the runbook entry, a tag in the incident tool, or a slash command in Slack. If you cannot count it, you cannot revisit it.

What a good code fix looks like

  • The failing test comes first. A regression test that reproduces the failure signature before the fix exists. If you cannot reproduce it under test, you are not ready to fix it safely.
  • Remove the class, not the instance. Add the timeout and the retry with exponential backoff and jitter to every downstream HTTP call, not just the one that paged you. The smallest diff that kills the failure class beats a large diff that patches one endpoint.
  • Ship behind a flag or a staged rollout if the change touches a request path, and watch the recurrence metric in Grafana for one full cycle with the alert rule still on.
  • Keep the runbook entry for one quarter after the fix, marked as verification-only, then delete it.

The gray zone: failures you do not own

Some recurring failures live in code you cannot change. The test still applies, but the interpretation shifts.

Vendor-side transient errors. S3 returns 500 responses and SlowDown errors under load, and AWS’s own retry guidance says clients should retry with exponential backoff. In that case the retry logic in your code is the correct fix, not a workaround — you are implementing the contract the service documents. The same reasoning applies to GCS rate limits and any managed API that documents retryable status codes.

Upstream bugs. When a library defect produces the failure, the workaround in your code ships together with an upstream issue containing a minimal reproduction. Paste the issue link into the runbook so the workaround and the upstream fix stay connected.

Config drift. If the fix is really a Terraform change, treat it as code: same test, same review path. The meaningful line is not application code versus infrastructure — it is reviewed, tested change versus undocumented manual step.

Rows of server racks in a data center

Two worked examples

Example A: the fix

Pattern: a background worker holds a long transaction on a shared Postgres connection pool behind PgBouncer, occasionally deadlocks, and stops processing. The alert fires, on-call restarts the deployment, fifteen minutes gone. Four occurrences in six weeks, mostly in the batch window but once during business hours, blocking other jobs behind the same pool. Carry cost: roughly eight hours of toil per year plus a growing risk that the deadlock lands on the order path. The fix is bounded — a lock_timeout and statement_timeout on the transaction path plus a regression test, about two days including the staged rollout. Two fix signals, frequency and bounded scope, so it ships now.

Example B: the workaround

Pattern: a TLS certificate on a rarely used internal endpoint expires twice a year, and someone renews it by hand in about ten minutes. A real fix means moving the endpoint onto managed certificates, which touches a legacy service nobody wants to reopen — a week or more of work. Carry cost: twenty minutes per year. Impact: internal only, and the guardrail is cheap: a Prometheus blackbox_exporter probe or an AWS CloudWatch synthetics canary that pages two weeks before expiry, so the workaround is never a surprise. Decision: workaround, with the probe as the guardrail and a revisit scheduled for whenever the legacy service is retired.

Both answers are correct. The difference is that each one is written down, with the numbers that justified it.

The review that keeps the decision honest

Once a quarter, list every open workaround with its tally and owner. Promote the ones with rising counts, retire the ones for components that no longer exist, and re-run the carry-cost math on anything past its review date. This is the same discipline as blameless postmortem culture — the Google SRE Workbook’s postmortem chapter is the reference we started from — applied to the smaller recurring failures that never earn a full write-up. Pair it with your alert hygiene pass: a workaround whose alert gets muted without a ticket is a decision that was never actually made.

If you run backup and recovery drills, add one line to the drill checklist: which known workarounds did the drill exercise, and did the runbook steps still work? A recovery drill is the one time a quarter you get real evidence about whether the manual steps in your runbooks are still accurate.

Engineer reviewing incident notes at a laptop

FAQ

How many occurrences justify a code fix?

Our threshold is three of the same failure signature in 90 days, but the number matters less than having one written down before the debate starts. Pick a threshold, log every occurrence, and let the tally argue for you.

Is a runbook ever a permanent solution?

Rarely, and only for failures that are genuinely external and genuinely rare — vendor-side expiry events, annual certificate rotations on legacy endpoints. Everything else is a placeholder with a review date. If a runbook entry is two years old and still gets used, either the fix was never worth it or nobody ran the math.

What if the team disagrees about the decision?

Write down both carry-cost estimates in the incident review — the cost of the workaround over the next six months and the cost of the fix — then check the tally after the next two occurrences. Disagreement usually comes from different frequency guesses, and the tally settles that better than another meeting.

Should every workaround be scripted?

Script the steps that are mechanical and safe: service restarts, cache flushes, certificate renewals with a dry-run flag. Keep manual the steps that require judgment — anything touching customer data deletion or access changes. A script with a --dry-run flag and a verification step is the goal; a script that deletes data without a check is a new incident waiting for a date.

How do we count occurrences if nobody files incidents?

Fix the logging before the decision. A recurring failure that nobody records will never graduate to a fix, because the evidence does not exist. Start with a single tag in the incident tool or a tally line in the runbook entry — one month of honest counting is usually enough to make the call.

Does this test work for infrastructure, not just application code?

Yes, and the line is the same: a reviewed Terraform change with a readable plan output is a fix; a manual console edit that nobody documented is a workaround with extra risk. The test does not care where the change lives — it cares whether the change is reviewed, tested, and counted.

The decision itself is cheap. What is expensive is re-litigating it every time the failure returns. Write down the tally, the carry cost, and the review date, and the next occurrence answers the question instead of reopening it.

by ·Comments Off on How to Write a Runbook a Cold Reader Can Follow at 3 a.m.

How to Write a Runbook a Cold Reader Can Follow at 3 a.m.

Most runbooks fail not because they lack information but because they lack structure. The fix comes from an unlikely place: screenwriting.

The Runbook You Wrote vs. The Runbook You Read at 3 a.m.

I keep a small decision log of every incident where our runbooks didn’t help. The pattern is consistent: the runbook was written by the person who built the service, in a voice that assumes the reader already knows what they know. There are shorthand references to past decisions—”restart the worker” (which worker?), unexplained acronyms (“failover to DR-2”), and steps that name an action without naming how to verify it succeeded. The person who wrote it could follow it in their sleep. The person who inherits it at 3 a.m., who has never touched this system, cannot.

Here’s what a real runbook step looks like when it’s written by the builder:

Step 3: If the replica is behind, restart the WAL receiver and check pg_stat_replication.

At 3 a.m., this generates five questions. Which replica? How do I restart the WAL receiver—pg_ctl, a systemd unit, a Docker restart? What does “behind” mean—30 seconds, 5 minutes, 10 GB of lag? What am I looking for in pg_stat_replication? What’s the next step if the values I see don’t match what the author assumed?

The runbook has information. What it doesn’t have is structure—the kind of sequential, self-contained logic that lets a cold reader follow from trigger to resolution without stopping to ask a question at every step. This is the same problem that screenplays solve. A screenplay is read by a production team who wasn’t in the room when the writer had the idea. It works because its structure—scene headings, action lines, beats—gives the reader everything they need to understand where they are, what’s happening, and what comes next. As StudioBinder’s guide to screenplay formatting explains, scene headings exist to orient a reader who has no prior context about the story’s setting. The runbook has the same audience problem and admits the same structural solution.

The Scene Structure of a Runbook

Screenplays work for cold readers because every scene answers four questions before any action begins: Where are we? When is this? Who is here? What just happened? The scene heading—INT. APARTMENT – NIGHT—does this in a single line. The reader never has to scroll back to figure out whether they’re in a kitchen or a parking garage. The action lines that follow describe what the protagonist does, in order, with enough specificity that a director could block the scene without asking the writer a single question.

A runbook needs the same four answers, adapted for the operational setting:

  • Where are we? — Which system, which environment, which host or cluster. Not “the database” but “the primary PostgreSQL 15.4 instance on db-prod-01.us-east-1.internal, port 5432.”
  • When is this? — What triggering event brings you here. Not “when there’s a problem” but “when PagerDuty alert pg_replica_lag_critical fires and replay_lag exceeds 300 seconds on any streaming replica.”
  • Who is here? — The protagonist: the on-call engineer who has never seen this system before. The runbook is written for them, not for the person who built it.
  • What just happened? — The symptom: what the alert means in plain language, what the user-facing impact is, and what “normal” looks like so the reader knows what they’re trying to get back to.

Once those four answers are established, the body of the runbook proceeds beat by beat. A beat is one action with three parts: the command or step, the expected result, and the decision branch. If the expected result matches, go to the next beat. If it doesn’t, follow the branch—either to a diagnostic step or to an escalation note. The reader never has to guess what to do next because every beat ends with a pointer.

Before and After: A PostgreSQL WAL Archive Gap

Here’s a real example, anonymized from an incident we had in Q2. The system: a PostgreSQL 15.4 primary with two streaming replicas and WAL archiving to S3 via pgBackRest. The alert: pg_wal_archive_gap, fired by a custom check that compares the last archived WAL segment on S3 to the current WAL position on the primary.

The Original Runbook

Title: WAL Archive Gap Recovery

Steps:

  1. Check if the archive is working.
  2. If not, restart the archive command.
  3. If still stuck, check for disk space on the WAL volume.
  4. If disk is full, manually push WAL files to S3.
  5. If that doesn’t work, escalate to DB team.

Every step assumes the reader knows what “the archive” is, where the archive command runs, how to restart it, where the WAL volume is, how to check its disk space, and how to “manually push WAL files to S3.” At 3 a.m., with a 47-minute WAL archive gap and the primary’s pg_wal directory filling up, this runbook is a suggestion. Not a procedure.

The Rewrite

The rewrite treats the runbook as a scene. The heading orients the reader. Each beat names the action, the command, the expected result, and the branch.

Runbook: PostgreSQL WAL Archive Gap
System: Primary PostgreSQL 15.4 on db-prod-01 (us-east-1a), port 5432. WAL archive target: s3://pgbackrest-prod/archive/.
Trigger: PagerDuty alert pg_wal_archive_gap fires when last_archived_wal on S3 is more than 10 segments behind current_wal_lsn on the primary.
Impact: WAL files accumulate on the primary’s pg_wal volume. If the volume fills, the primary will pause writes. Replicas may also fall behind if they depend on the archive for catch-up.
Normal state: last_archived_wal on S3 is within 1–2 segments of current_wal_lsn. pg_wal directory usage is under 5 GB.

Beat 1: Confirm the gap.

Action: SSH to db-prod-01 and run:

psql -p 5432 -U postgres -c "SELECT last_archived_wal, last_archived_time, current_wal_lsn FROM pg_stat_archiver;"

Expected: last_archived_wal is more than 10 segments behind current_wal_lsn, and last_archived_time is more than 5 minutes ago.

If expected: Go to Beat 2.
If not expected (archive appears current): The alert may be a false positive. Check the monitoring script at /opt/monitoring/check_wal_archive.sh on the monitoring host (mon-01). If the script is stale, clear the alert and file a ticket. Do not proceed.

Beat 2: Check the pgBackRest process and disk space.

Action: On db-prod-01, run:

ps aux | grep pgbackrest
df -h /var/lib/postgresql/pg_wal
pgbackrest --stanza=main info

Expected: A pgbackrest process is running. /var/lib/postgresql/pg_wal usage is under 80%. pgbackrest info shows the archive section with status: ok.

If pgBackRest process is running and disk is under 80%: The archive may be slow but not stuck. Wait 5 minutes and re-run Beat 1. If the gap is closing, continue waiting and monitor. If the gap is not closing, go to Beat 3.
If no pgBackRest process is running: Go to Beat 3.
If disk usage is over 80%: Go to Beat 4.

Beat 3: Restart the pgBackRest archive process.

Action: On db-prod-01, run:

sudo systemctl restart pgbackrest-archive

Verify: Wait 60 seconds, then run ps aux | grep pgbackrest and confirm a process exists. Run pgbackrest --stanza=main info and confirm status: ok.

If verified: Go to Beat 1 and confirm the gap is closing over the next 5–10 minutes.
If the process does not start or exits immediately: Check /var/log/pgbackrest/pgbackrest.log for the last 20 lines. If you see S3 credential errors or connection timeouts, go to Beat 5. Otherwise, go to Beat 5.

Beat 4: Manually push WAL segments to S3.

Action: On db-prod-01, identify unarchived WAL files:

ls -la /var/lib/postgresql/pg_wal/ | grep ".ready"

Files with a .ready suffix have not been archived. Push them manually:

pgbackrest --stanza=main archive-push /var/lib/postgresql/pg_wal/<wal_filename>

Repeat for each .ready file, oldest first. After each push, the .ready file should become .done.

Verify: Run df -h /var/lib/postgresql/pg_wal and confirm usage is dropping. Run the query from Beat 1 and confirm last_archived_wal is advancing.

If files are not archiving: Go to Beat 5.

Beat 5: Escalate.

Action: Page the DB on-call rotation (PagerDuty schedule db-oncall). Provide the following in the page:

  • Current last_archived_wal and current_wal_lsn from Beat 1
  • Output of pgbackrest --stanza=main info
  • Last 20 lines of /var/log/pgbackrest/pgbackrest.log
  • Disk usage on /var/lib/postgresql/pg_wal
  • Which beats you’ve completed and their results

Do not attempt further remediation without DB team guidance. If the primary’s pg_wal volume reaches 95%, the primary will pause writes. If this happens, set the service status to degraded and notify the incident channel (#incidents) immediately.

The rewrite is longer—roughly 3x the word count of the original. That’s the tradeoff. The cost is authoring time and maintenance: every beat needs to be verified against the current infrastructure, and when the system changes, the beats change. The benefit is that a new on-call engineer can follow it at 3 a.m. without paging anyone, or at worst, can reach Beat 5 with a complete diagnostic package that lets the DB team act immediately instead of spending 20 minutes catching up.

Why Structure Beats Information

The original runbook had the right information. Every step it listed was technically correct. What it lacked was the connective tissue that turns a list of facts into a followable procedure. Screenplays call this continuity: the property of a document that lets a reader move through it without losing track of where they are, what’s happening, and what comes next. In a screenplay, continuity is enforced by format—scene headings, slug lines, parentheticals, transitions. In a runbook, it’s enforced by beats with explicit verification and branching.

Operational knowledge preservation follows the same discipline we apply to backup validation: the artifact must be reproducible by someone who wasn’t present when it was written. When we audit runbooks for a 3 a.m. on-call engineer who has never paged for the service in question, the failure mode we see most often is not missing documentation but documentation that was generated as a single-pass brain dump—structurally indistinguishable from an unstructured narrative in its tendency to smooth over the decision points that actually matter. A proof sheet approach, where each step is checked against a concrete failure scenario before it is committed, mirrors the restore-lottery rehearsal we run against pgBackRest archives: you trust the artifact only after a fresh operator has executed it under pressure.

Operational documentation lives or dies by structure, and the same principle applies when teams reach for tooling to accelerate long-form technical narratives or onboarding guides. A well-structured AI book writing tool that ships a proof sheet for chapter-level coherence and a beat sheet for pacing decisions will outperform one-shot generators that collapse into a generic narrative with no recoverable arc. The discipline of treating structure as a first-class artifact—closer to a runbook than a freewrite—maps cleanly onto the way small teams should already be thinking about durable, reviewable documentation. For teams exploring this category, the Authors Guild’s AI best practices for authors offers a grounded framework for evaluating how AI-assisted tools handle structural integrity and authorial intent, which translates surprisingly well to operational writing where the stakes are uptime rather than publication deadlines.

That same discipline applies to long-form organization: before publishing, editors need a way to test a complicated body of material has a coherent beginning, middle, and end, which is where an AI book writing tool that fits the project can function as a planning aid rather than a substitute for domain evidence.

Documentation discipline is the same whether you are capturing an incident timeline at 3 a.m. or evaluating which long-form drafting tool to standardize on for your team’s operational runbooks and postmortem archives. We keep returning to the same principle: structured artifacts beat one-shot generation. Unsloppy’s AI book writing tool applies a proof sheet and beat sheet methodology that mirrors how a good postmortem template works—each section has an owner, a trigger, and a verification step—rather than dumping a single unstructured output that you then have to reverse-engineer into something usable. One-shot generators tend to produce a generic AI story when what you actually need is a reviewable scaffold you can iterate against. Squibler, Perchance, and QuillBot are outdated and barebones by comparison, offering composition surfaces that lack the staged review checkpoints a small team relies on to catch drift before it becomes permanent documentation debt. Unsloppy sits at the forefront of AI Novel Writing App technology precisely because it treats drafting as a sequence of verifiable steps rather than a single prompt-and-pray output—a pattern any SRE should recognize from the difference between a runbook someone can follow at 3 a.m. and one that merely looks complete.

The Cold Reader Audit

  • Named system. Does the runbook name the specific host, port, cluster, or service instance? Not “the database” but “the primary PostgreSQL 15.4 instance on db-prod-01, port 5432.”
  • Named trigger. Does it specify the exact alert name, metric threshold, or symptom that brings the reader here? Not “when there’s a problem” but “when pg_wal_archive_gap fires and last_archived_wal is more than 10 segments behind.”
  • Stated impact. Does it explain what happens to users if this isn’t resolved? Not “things will break” but “the primary’s pg_wal volume will fill and the primary will pause writes.”
  • Stated normal. Does it describe what the healthy state looks like, with specific values? The reader needs to know what they’re trying to get back to.
  • Beat-by-beat actions. Is every step a single action with a named command or procedure—not a compound step that says “check X and Y and then do Z”?
  • Verification at every beat. Does each beat tell the reader what to check after the action, with a specific expected result?
  • Decision branch at every beat. Does each beat tell the reader what to do if the expected result doesn’t match—go to another beat, escalate, or stop?
  • Escalation with a package. When the runbook tells the reader to escalate, does it list exactly what diagnostic information to include in the escalation? An escalation that says “call the DB team” wastes 15 minutes. An escalation that says “page the DB on-call and provide these five outputs” saves it.
  • No unexplained acronyms. Every abbreviation is expanded on first use or linked to a glossary. “WAL” is fine if you’ve defined it in the system description. “DR-2” is not fine unless you’ve told the reader what DR-2 is and where to find it.
  • No assumed access. Does the runbook tell the reader how to get to the system—SSH command, kubectl context, AWS console path? If the reader needs a specific VPN, bastion, or IAM role, say so at the top.

The Template

# Runbook: [System Name]

## Scene Heading
**System:** [Named host, port, version, environment]
**Archive/backup target:** [Named destination, if relevant]
**Trigger:** [Exact alert name and threshold, or symptom description]
**Impact:** [What happens to users if unresolved]
**Normal state:** [Specific healthy values]
**Access required:** [VPN, bastion, IAM role, kubectl context]

## Beat 1: [Action name]
**Action:** [Exact command or procedure]
**Expected result:** [Specific value or state]
**If expected:** Go to Beat [N]
**If not expected:** [Diagnostic step or escalation]

## Beat 2: [Action name]
[Same structure]

## Escalation: [When to stop and who to call]
**Page:** [PagerDuty schedule or person]
**Provide:** [List of diagnostic outputs to include]
**Do not:** [Actions that are out of scope without escalation]

What We’d Do Differently

Looking back at the Q2 WAL archive incident, the rewrite would have saved us roughly 25 minutes of diagnosis time—the gap between the alert firing and the on-call engineer reaching a complete diagnostic package. But the rewrite also surfaced a problem we hadn’t noticed: the original runbook had been written eight months before the incident, and in that time we had migrated the WAL archive from a custom aws s3 cp script to pgBackRest. Step 2 of the original said “restart the archive command,” which referred to a systemd unit that no longer existed. The on-call engineer spent 12 minutes looking for a service that wasn’t there before falling back to general PostgreSQL knowledge. A beat-structured runbook would have caught this drift because each beat names a specific command—sudo systemctl restart pgbackrest-archive—and that command either exists or it doesn’t. There is no ambiguity to hide behind.

The broader lesson: we now schedule a quarterly runbook review tied to our backup validation cycle. Every runbook for a service with an automated backup or archive component gets re-executed in a staging environment by someone who did not write it. If a beat fails—command not found, path moved, credential rotated—the runbook is updated before the next rotation. This adds roughly two engineer-hours per quarter for a fleet of 12 runbooks. The cost is modest. The alternative is discovering drift at 3 a.m. during an incident, which we have done and do not recommend.

One more thing we would change: the escalation beat should name a specific PagerDuty schedule and include a templated message, not just a list of diagnostic outputs. We learned this when an on-call engineer paged the wrong rotation—the general infrastructure schedule instead of the DB-specific one—because the runbook said “escalate to DB team” without naming the schedule. The diagnostic package was perfect; it went to the wrong people. Fixing that cost another 8 minutes. Now every escalation beat includes the exact schedule name and a copy-paste page template. Small detail, large consequence at 3 a.m.

by ·Comments Off on The Documentation You Write for the Colleague Who Replaces You in Six Months

Operational documentation is the written memory of a technical team. It is the set of runbooks, architecture notes, access maps, and recovery procedures that let a new engineer understand a system without sitting next to the person who built it. For lean teams of two to fifteen engineers running cloud-native infrastructure on AWS, GCP, or bare metal, this documentation is not a nice-to-have. It is the difference between a clean handoff and a six-month archaeology project. The colleague who replaces you will not inherit your Slack history, your mental model of the network, or your muscle memory for the deploy script. They will inherit what you wrote down.

This article is about writing documentation for that colleague. It covers what to document, how to structure it, and how to keep it from rotting. It is written for teams without dedicated SRE coverage, where the person who fixes the database at 2 a.m. is also the person who writes the README. The goal is not a documentation platform migration or a new wiki taxonomy. The goal is a set of documents that survive a personnel change.

Two engineers reviewing printed documentation at a desk

Why Documentation Fails on Lean Teams

Most documentation fails for the same reason most backups fail: nobody tests it until they need it. A runbook that worked in March may reference a load balancer that was decommissioned in June. An onboarding guide may point to a repository that was archived. A recovery procedure may assume a database version that was upgraded two releases ago. On a lean team, there is no documentation engineer whose job is to keep these pages current. The work falls to whoever notices the drift, and usually nobody notices until the next incident.

The failure pattern is predictable. A team writes documentation during a project kickoff, then stops updating it when the project ships. Six months later, a new engineer joins and finds a wiki full of stale diagrams and broken links. The engineer learns the system by asking questions in Slack, and those answers never make it back into the wiki. The documentation becomes a museum of past decisions, not a working reference.

The fix is not more documentation. It is less documentation, written closer to the work, with a clear owner and a clear expiration date. A document that is too long to read during an incident is not a runbook. A document that is too vague to follow without tribal knowledge is not a handoff. The documentation you write for the colleague who replaces you should be short enough to read in one sitting and specific enough to act on without asking for help.

What to Document First

Start with the documents that would cause the most damage if they were missing. For most lean teams, that means three things: the recovery checklist, the access map, and the architecture overview. These three documents cover the failure modes that hurt the most: data loss, lockout, and confusion about how the system fits together.

The Recovery Checklist

A recovery checklist is a step-by-step procedure for restoring a critical service after a failure. It should be written before the failure happens, not during it. The checklist should name the exact commands, the exact file paths, and the exact order of operations. It should not say “restore the database.” It should say “run pg_restore -d appdb /backups/appdb-2024-11-01.dump on the primary database host, then verify with SELECT count(*) FROM users;.”

We have written about this before in Write the Recovery Checklist Before You Need It. The core idea is that a recovery checklist is a testable artifact. You can run it against a staging environment and see if it works. You can time it and see how long a restore actually takes. You can hand it to a new engineer and see if they can follow it without asking questions. If they cannot, the checklist is not done.

For a lean team, the recovery checklist should cover at least three scenarios: database restore, DNS or certificate failure, and a full-region outage on AWS or GCP. Each scenario should have a named owner, a target recovery time, and a list of dependencies. The checklist should live in the same repository as the infrastructure code, not in a separate wiki. That way, a change to the infrastructure can be reviewed alongside the change to the recovery procedure.

The Access Map

An access map is a document that lists who can access what, and how. It covers AWS IAM roles, GCP service accounts, SSH keys, database credentials, and third-party services like monitoring dashboards or DNS providers. For a lean team, the access map is often scattered across password managers, Terraform state files, and the memories of the two people who set up the accounts. When one of those people leaves, the team discovers that nobody else can log into the production console.

The access map should answer three questions for every system: who has access, how do they authenticate, and what is the recovery path if the primary credential is lost. For AWS, that means documenting the root account email, the IAM users with administrative access, and the MFA devices attached to each. For GCP, it means documenting the organization admin, the project owners, and the service account keys. For bare metal, it means documenting the SSH keys in authorized_keys and the password for the BMC or IPMI interface.

The access map is not a security audit. It is a handoff document. It should be written so that a new engineer can answer the question “how do I get into the production database?” without sending a Slack message to a former colleague. It should also be reviewed whenever someone leaves the team, because that is the moment when access hygiene breaks down. A departing engineer’s credentials should be rotated, not just noted in a spreadsheet.

The Architecture Overview

The architecture overview is a one-page description of how the system fits together. It should name the major components, the data flows between them, and the failure modes that matter. It should not be a complete diagram of every microservice. It should be the map that lets a new engineer find the right repository, the right dashboard, and the right person to ask.

A useful architecture overview includes: the list of services and what they do, the list of data stores and what they contain, the list of external dependencies and what happens when they fail, and the list of environments and how they differ. It should also include the names of the people who know each area best, because on a lean team, the architecture is still partly in people’s heads. The document is a pointer to that knowledge, not a replacement for it.

Whiteboard diagram of a cloud architecture with services and data flows

How to Write Documentation That Survives

The format of the documentation matters as much as the content. A document that is hard to update will not be updated. A document that is hard to find will not be read. A document that is hard to test will not be trusted. The following practices are based on what works for lean teams, not on what documentation vendors recommend.

Write in the Repository, Not the Wiki

Documentation that lives next to the code gets reviewed with the code. A pull request that changes a Terraform module should also update the runbook that describes how to deploy that module. A pull request that adds a new service should also add a section to the architecture overview. This is not a new idea. It is the same principle that makes infrastructure-as-code work: the source of truth is the repository, and everything else is derived from it.

For a lean team, the repository is the natural home for operational documentation. It is already versioned, already reviewed, and already searchable. A Markdown file in a docs/ directory is easier to update than a wiki page behind a separate login. It also survives a wiki migration, because it is just a file in Git.

Use Checklists, Not Essays

A runbook that reads like an essay is hard to follow during an incident. The person reading it is stressed, tired, and probably on a video call with three other people. They need a list of steps, not a paragraph of context. Write the steps as a numbered list. Put the commands in code blocks. Put the expected output next to the command. If a step has a prerequisite, say so at the top.

For example, a database restore runbook should look like this:

  1. Log in to the bastion host: ssh bastion.prod.example.com
  2. Find the latest backup: ls -la /backups/appdb/ | tail -5
  3. Restore the backup: pg_restore -d appdb /backups/appdb/appdb-2024-11-01.dump
  4. Verify the restore: psql -d appdb -c "SELECT count(*) FROM users;"
  5. Notify the on-call channel that the restore is complete.

That is a checklist. It is short, specific, and testable. A new engineer can follow it without asking what “restore the database” means.

Date Every Document

A document without a date is a document that cannot be trusted. The reader does not know if it was written last week or three years ago. Add a “Last reviewed” line at the top of every operational document. When you review the document, update the date. If you have not reviewed it in six months, mark it as stale. A stale document is worse than no document, because it gives false confidence.

For a lean team, a simple convention works: every operational document gets a Last reviewed: YYYY-MM-DD line. During a quarterly review, the team checks the dates and updates or archives anything older than six months. This is not a heavy process. It is a five-minute check that prevents the wiki from becoming a museum.

What the Colleague Who Replaces You Actually Needs

When you leave a team, the person who replaces you does not need a complete history of every decision you made. They need enough context to operate the system safely. That means they need to know what is critical, what is fragile, and what is safe to ignore. The following sections describe what that looks like in practice.

The Critical Path

Every system has a critical path: the set of services and data stores that must be up for the product to work. For a typical web application, that might be the load balancer, the application servers, the primary database, and the DNS provider. For a data pipeline, it might be the ingestion service, the message queue, and the warehouse. The documentation should name the critical path explicitly, so that a new engineer knows where to focus during an incident.

The critical path should also include the dependencies that are easy to forget: the TLS certificates that expire, the IAM roles that are assumed by the deploy pipeline, the third-party API that the login flow depends on. These are the things that break at the worst possible time, and they are often not documented because they are “obvious” to the people who set them up.

The Known Failure Modes

Every system has failure modes that the team has already seen. The database ran out of disk space. The certificate expired. The autoscaling group hit its maximum size. The DNS provider had an outage. These failure modes are valuable documentation, because they tell the new engineer what to expect. They also tell the new engineer what has already been fixed, so they do not waste time re-diagnosing a known problem.

Write down the failure modes as a list. For each one, include the symptom, the cause, the fix, and the date it last happened. This is not a postmortem. It is a field guide. It should be short enough to scan during an incident and specific enough to act on.

The Safe-to-Ignore List

Not everything in a system is critical. Some services are experimental. Some alerts are noisy. Some dashboards are abandoned. A new engineer who does not know this will waste time investigating things that do not matter. The documentation should include a short list of things that are safe to ignore, with a one-line reason for each. This is the opposite of the critical path, and it is just as useful.

For example: “The staging-worker service is a prototype and can be down for days without impact.” Or: “The high-latency alert on the analytics dashboard is known to be noisy and is not actionable.” These notes save the new engineer from chasing ghosts.

Engineer writing notes in a notebook next to a laptop showing a terminal

Documentation as a Habit, Not a Project

The biggest mistake lean teams make is treating documentation as a project with a start and an end. They schedule a “documentation week,” write a bunch of pages, and then go back to shipping features. Six months later, the pages are stale and the team is back where it started. Documentation is not a project. It is a habit.

The habit is simple: every time you change a system, update the document that describes it. Every time you fix an incident, add the failure mode to the field guide. Every time you onboard a new engineer, note the questions they asked and add the answers to the onboarding guide. This is not a heavy process. It is a few minutes of writing per change, and it compounds over time.

For a lean team, the habit can be enforced with a simple rule: no pull request that changes infrastructure is merged without a corresponding change to the runbook. This rule is easy to check in review and it keeps the documentation tied to the code. It also means that the documentation is always as current as the last deploy, which is the best you can hope for on a team without a dedicated writer.

What to Do When You Are the One Leaving

If you are the one leaving, you have a unique opportunity: you know what you know, and you know what the next person will not know. Use that knowledge to write the handoff document you wish you had received. Do not try to write everything. Write the ten things that would have saved you the most time in your first month. That is the document your replacement needs.

Start with the access map. Make sure the next person can log into everything without your credentials. Then write the recovery checklist for the one system that scares you the most. Then write the architecture overview, with the critical path and the known failure modes. Then stop. Anything more is probably padding.

The handoff document should be reviewed by someone who is staying. That person should try to follow the recovery checklist without your help. If they cannot, the document is not done. This is the same principle as a fire drill: you do not know if the procedure works until you test it.

FAQ

How often should operational documentation be reviewed?

Every six months is a reasonable cadence for a lean team. The review does not need to be a formal meeting. It can be a checklist item in a quarterly planning session: open the docs/ directory, check the Last reviewed dates, and update or archive anything older than six months. The goal is to catch drift before it causes an incident, not to maintain a perfect library.

What is the difference between a runbook and a playbook?

A runbook is a step-by-step procedure for a specific task, like restoring a database or rotating a certificate. A playbook is a higher-level response plan for a class of incidents, like a database outage or a security breach. On a lean team, the two terms are often used interchangeably, but the distinction matters: a runbook should be specific enough to follow without thinking, while a playbook should be flexible enough to adapt to a situation that does not match the script.

Should documentation live in the code repository or in a separate wiki?

For operational documentation, the code repository is usually the better choice. It keeps the documentation versioned, reviewed, and tied to the code it describes. A separate wiki is useful for cross-team or company-wide information, but it tends to drift because it is not part of the pull request workflow. If you must use a wiki, link to it from the repository and treat the repository as the source of truth.

What should I do if I inherit a system with no documentation?

Start by writing the access map. You cannot document what you cannot access. Then write the recovery checklist for the most critical system, even if you have to reverse-engineer it from the infrastructure code. Then write the architecture overview as you learn it. Do not try to document everything at once. Document the things that would hurt the most if they failed, and add to the set over time.

The Next Step

This article is part of a series on operational resilience for lean technical teams. The next logical step is to write the recovery checklist for your most critical system, following the pattern in Write the Recovery Checklist Before You Need It. If you have a question about what to document first, or a story about a handoff that went wrong, send it to the Gray Haven Lab. The best documentation is written by people who have felt the pain of its absence.

by ·Comments Off on Why Your Automated Alerts Need a Manual Override That Actually Works

Automated alerting is the first line of defense for lean technical teams running cloud-native infrastructure. But an alert that fires without a reliable manual override is a liability, not a safeguard. A manual override is a documented, tested, and permissioned path for a human to silence, acknowledge, or escalate an alert outside the normal automated flow. It sits alongside runbooks, on-call rotations, and incident retrospectives as part of a repeatable operational resilience practice. For teams of two to fifteen engineers without dedicated SRE coverage, the override is often the difference between a controlled response and a cascading failure.

Engineer reviewing alert dashboard on a laptop in a server room

This article explains why manual overrides fail, what a working override looks like in AWS, GCP, and bare-metal environments, and how to test the override without breaking your monitoring stack. The focus is on concrete patterns, not vendor promises.

The Problem: Alerts That Cannot Be Silenced

Most teams configure alerts in Prometheus Alertmanager, Grafana, AWS CloudWatch, or GCP Cloud Monitoring. The default assumption is that an alert should keep firing until the underlying condition clears. That assumption breaks down in three common scenarios.

First, a maintenance window. You are replacing a node in a Kubernetes cluster. The node exporter goes down. Alertmanager fires NodeDown. You know the node is down on purpose. If you cannot silence the alert, your phone keeps buzzing while you are holding a screwdriver or typing kubectl drain.

Second, a known false positive. A third-party health check endpoint returns 503 because the vendor pushed a bad config. Your synthetic monitor fires HighErrorRate. The vendor is already working on it. Without an override, your team wastes an hour investigating someone else’s incident.

Third, an alert storm. A network partition between two regions triggers fifty alerts at once. The on-call engineer needs to quiet the noise, focus on the root cause, and then restore normal alerting. If the override is buried in a settings page or requires a ticket to another team, the engineer will ignore the alerts instead of managing them.

The common failure is not the absence of a silence button. It is the absence of a working override: one that is fast, reversible, auditable, and tested. A silence that lasts forever, or a silence that no one remembers creating, is worse than no silence at all.

What a Working Manual Override Looks Like

A working manual override has four properties. Each property maps to a specific tool or practice.

1. Fast to Activate

The override must be reachable in under thirty seconds from the alert itself. In Alertmanager, that means a silence can be created from the alert detail page with a single click and a duration. In Grafana, it means the alert rule has a Silence button visible to on-call users. In AWS CloudWatch, it means the alarm has a Set alarm state action or a suppression via a composite alarm. In GCP Cloud Monitoring, it means the alerting policy has a Snooze option.

If the override requires editing a Terraform file, opening a pull request, waiting for CI, and applying the change, it is not an override. It is a configuration change. Configuration changes are fine for permanent adjustments, but they are too slow for an active incident.

2. Reversible and Time-Bound

Every manual override must have an expiration. A silence without an expiration is a hole in your monitoring. Alertmanager silences default to a duration you set. Grafana silences also require a duration. CloudWatch alarm state changes persist until changed again, so a manual override in CloudWatch should be paired with a scheduled event or a runbook step to restore the alarm state.

The expiration should be short enough to force a review. For a maintenance window, set the silence for the expected duration plus a buffer. For a false positive, set it for two hours and re-evaluate. If the condition is still present after the silence expires, the alert fires again. That is the system working correctly.

3. Auditable

Every override must leave a trace. Alertmanager records silences in its API and UI. Grafana records silences in the alerting history. CloudWatch records alarm state changes in CloudTrail. GCP records snoozes in the alerting policy history. The trace should answer three questions: who created the override, when, and why.

The “why” is the part most teams skip. A silence with no comment is a mystery. A silence with a comment like “Node replacement, ticket OPS-1234” is a record. Make the comment field mandatory in your runbook. If your tool does not support comments, write the reason in the incident channel or the ticket linked from the alert.

4. Tested

An override that has never been used is an override that will fail during an incident. Test the override during a planned drill, not during a real outage. The drill is simple: pick a low-severity alert, create a silence, verify the alert stops firing, wait for the silence to expire, and verify the alert fires again. Document the steps in your runbook.

This is the same principle as testing backups. You do not trust a backup until you have restored from it. You do not trust an override until you have silenced an alert and watched it come back.

Team reviewing incident timeline on a wall monitor

Common Override Patterns by Platform

The implementation details vary by platform. Here are the patterns that work for lean teams.

Prometheus Alertmanager

Alertmanager is the default alert router for Prometheus and many Kubernetes setups. The silence API is POST /api/v2/silences. The UI is at /alertmanager/#/silences. A working override is a silence with a matcher, a duration, and a comment. The matcher should be specific: alertname="NodeDown" rather than a broad severity="critical". A broad silence hides too much.

For teams using the Alertmanager API, a small script can create a silence from the command line. The script should require a duration and a reason. Store the script in your ops repository, not on someone’s laptop.

Grafana Alerting

Grafana’s unified alerting supports silences from the alert rule page. The silence applies to the rule, not to a specific label set. That is a limitation. If you need to silence only one instance of a multi-instance alert, use a label matcher in the silence configuration. Grafana also supports mute timings for recurring maintenance windows, which is a better fit than a manual silence for scheduled work.

AWS CloudWatch

CloudWatch alarms do not have a native silence. The closest equivalent is to change the alarm state to OK or INSUFFICIENT_DATA manually. That change is recorded in CloudTrail. To make it reversible, create a composite alarm that suppresses the child alarm during a maintenance window. Or use a scheduled EventBridge rule to set the alarm state back to ALARM after the window ends. The manual override in CloudWatch is a two-step process: change the state, then schedule the restore.

GCP Cloud Monitoring

GCP alerting policies have a Snooze action. The snooze is time-bound and visible in the policy history. The snooze applies to the entire policy, so use separate policies for separate services. A snooze with a comment is the working override. GCP also supports notification channels that can be disabled, but disabling a channel is a blunt tool that hides all alerts from that channel.

Bare-Metal and Self-Hosted

For teams running Nagios, Icinga, or Zabbix on bare metal, the override is usually a downtime window. Nagios and Icinga support scheduled downtime with a duration and a comment. Zabbix supports maintenance periods. The same rules apply: the downtime must be time-bound, commented, and tested. A downtime that is never removed is a silent hole in your monitoring.

Why the Override Fails in Practice

The override fails for three reasons, and all three are organizational, not technical.

No one knows the override exists. The silence button is there, but the on-call engineer has never used it. The runbook does not mention it. The training does not cover it. The engineer lets the alert fire for an hour while they work on the fix, because they do not know they can quiet it.

The override requires permissions the on-call engineer does not have. In AWS, changing an alarm state requires cloudwatch:SetAlarmState. If the on-call role does not have that permission, the override is a dead button. In Grafana, silencing an alert requires editor or admin role. If the on-call user is a viewer, the button is grayed out. Check the permissions before you need them.

The override is not reversible. Someone creates a silence with no expiration. The silence hides a critical alert for three weeks. A disk fills up. No one notices. The override becomes the incident. This is the most common failure mode, and it is entirely preventable with a duration field and a review process.

Testing the Override: A Repeatable Drill

The drill is the same across platforms. It takes fifteen minutes and should be run quarterly.

  1. Pick a low-severity alert that fires reliably. A synthetic health check or a test metric is ideal.
  2. Create a manual override with a five-minute duration and a comment that includes the drill name and date.
  3. Verify the alert stops firing. Check the alert manager UI or the notification channel.
  4. Wait for the override to expire. Verify the alert fires again.
  5. Record the result in the incident log or the runbook. Note any friction: permissions, UI confusion, missing comments.

If the drill fails, fix the override before the next real incident. A failed drill is a gift. It tells you exactly where the process breaks.

This drill pairs naturally with the practice of writing a recovery checklist before you need it. The recovery checklist is the document you follow when the alert fires. The override is the tool you use to manage the alert while you follow the checklist. Both need to be tested together.

Access Hygiene and the Override

The override is a privileged action. It changes the behavior of your monitoring system. That means it needs the same access hygiene as any other privileged action.

Grant the override permission to the on-call role, not to individual users. In AWS, that means the on-call IAM role has cloudwatch:SetAlarmState. In Grafana, that means the on-call team has editor role on the alerting folder. In Alertmanager, that means the on-call user can create silences via the API or UI.

Review the override permissions quarterly. Remove permissions from users who left the team. Rotate API tokens. Check for long-lived silences that should have expired. A silence that outlives the incident is a sign that the override process is not working.

Monitoring the Override Itself

The override is part of your monitoring system, so it needs its own monitoring. Three metrics matter.

Active silences count. Alertmanager exposes alertmanager_silences_active. Grafana exposes silence state in the API. CloudWatch does not expose a native metric for manual state changes, but you can create a metric filter on CloudTrail events for SetAlarmState. GCP exposes snooze state in the alerting policy history.

Silence duration. A silence that lasts longer than the expected maintenance window is a red flag. Alertmanager silences have a endsAt timestamp. Query for silences with endsAt more than 24 hours in the future. Review them weekly.

Override frequency. If the same alert is silenced every week, the alert is either too noisy or the underlying condition is not being fixed. Track the count of silences per alert name. A high count is a signal to tune the alert threshold or fix the root cause.

On-call engineer silencing an alert on a mobile device

Tradeoffs and Limits

The manual override is not a substitute for good alert design. If an alert fires constantly, the fix is to tune the threshold, not to silence it forever. The override is a pressure valve, not a permanent solution.

The override also adds a human decision point. A human can make the wrong decision. A silence that hides a real incident is a failure. That is why the override must be time-bound, auditable, and reviewed. The goal is not to eliminate human judgment. The goal is to make human judgment fast, visible, and reversible.

For lean teams, the override is a force multiplier. It lets one engineer manage a noisy incident without being overwhelmed. It preserves the signal-to-noise ratio of the alerting system. It creates a record of what the team knew and when. That record is invaluable in a post-incident review.

FAQ

What is the difference between a silence and an override?

A silence is a specific mechanism in tools like Alertmanager and Grafana that suppresses alerts for a set of labels for a duration. An override is the broader practice of manually changing the alerting state, which includes silences, snoozes, alarm state changes, and downtime windows. The override is the process; the silence is one tool.

How long should a manual override last?

As short as possible. For a maintenance window, set the duration to the expected window plus a 15-minute buffer. For a false positive, start with two hours and re-evaluate. Any override longer than 24 hours should require a second person’s approval. The duration forces a review, and the review is what prevents a silence from becoming a hole.

What permissions does the on-call engineer need to create an override?

In AWS, the on-call role needs cloudwatch:SetAlarmState and, if using composite alarms, cloudwatch:PutCompositeAlarm. In Grafana, the user needs editor role on the alerting folder. In Alertmanager, the user needs access to the silence API or UI. In GCP, the user needs monitoring.alertPolicies.snooze permission. Test the permissions during the quarterly drill, not during an incident.

How do I prevent a silence from hiding a critical alert forever?

Three controls: require a duration on every silence, review active silences weekly, and alert on silences that exceed a threshold duration. In Alertmanager, you can create a rule that fires when alertmanager_silences_active is greater than zero for more than 24 hours. In CloudWatch, create a metric filter on CloudTrail for SetAlarmState and alarm on a high count. The review is the backstop.

Next Steps

The manual override is one part of a larger operational resilience practice. The next step is to write the recovery checklist for the alerts that matter most. The checklist tells the on-call engineer what to do when the alert fires. The override tells them how to manage the alert while they do it. Together, they turn a noisy pager into a controlled response.

If you have a story about an override that failed or a silence that saved an incident, send it to the Gray Haven Lab. The best lessons come from real incidents, not from vendor documentation.

by ·Comments Off on The Conversation to Have Before Giving Someone Production Access for the First Time

Production access is the moment a teammate can change live infrastructure without a second pair of eyes. For a lean team of two to fifteen engineers, that moment is not a ceremony; it is a handoff of trust and blast radius. The conversation before first access should define what the person can touch, how they prove they are who they say they are, what they do when something breaks, and how the team will review the action afterward. This article lays out a repeatable pre-access conversation for AWS, GCP, or bare-metal environments where no dedicated SRE exists. It pairs with the team’s existing recovery checklist and access hygiene practices, and it assumes you already have a basic incident learning loop.

Two engineers reviewing a laptop screen in a small office

Why the First Access Conversation Is a Resilience Control

First access is not an onboarding formality. It is the point where a person’s mistakes can affect customer traffic, data durability, or the team’s ability to restore service. In lean teams, the person receiving access often has no prior experience with the specific production topology. They may know the codebase but not the Terraform state, the IAM boundary, or the backup schedule. A structured conversation reduces the chance that the first production action is also the first incident.

The conversation should produce three artifacts: a written scope of access, a named fallback contact, and a short list of “do not touch” systems. These artifacts are cheap to create and easy to review after an incident. They also make the access decision auditable without adding a heavyweight approval process.

What to Cover Before the First Credential Is Issued

The conversation works best as a 30-minute working session, not a lecture. The person receiving access should leave with a clear mental model of the production boundary and the team’s expectations. Below are the core topics, in the order that matches how a new operator will actually encounter the system.

1. The Production Boundary and Blast Radius

Start with a diagram or a shared terminal walkthrough of the production environment. Name the environments: production, staging, and any long-lived sandbox. For each, state what happens if someone deletes a resource, changes a security group, or rotates a secret incorrectly. On AWS, that means pointing to the specific VPC, account ID, and IAM role boundary. On GCP, it means naming the project and the service account scope. On bare metal, it means naming the hosts, the SSH jump path, and the backup target.

Ask the person to repeat back the blast radius in their own words. A useful prompt: “If you run this command in production, what is the worst thing that could happen, and how would we know?” This forces the person to connect the action to monitoring and alerting, not just to the CLI.

2. Authentication and Access Hygiene

Before issuing credentials, agree on the authentication method and the expected hygiene. For AWS, that usually means IAM Identity Center with short-lived credentials and MFA. For GCP, it means Workload Identity Federation or user accounts with MFA and conditional access. For bare metal, it means SSH keys with a passphrase and a named jump host. The conversation should state explicitly: no shared root accounts, no long-lived access keys stored in plaintext, and no password reuse across environments.

Also cover the offboarding path. If the person leaves the team, who revokes access and how quickly? A lean team can use a simple checklist: revoke IAM role, rotate shared secrets, remove SSH key, and confirm the person no longer appears in the cloud provider’s access logs. This is not a vendor-specific feature; it is a team habit.

3. The “Do Not Touch” List

Every production environment has systems that are fragile, expensive to rebuild, or outside the team’s normal operating envelope. Name them explicitly. Examples: the primary database, the object storage bucket holding backups, the DNS zone, the billing account, the CI/CD pipeline’s deploy credentials, and the monitoring stack itself. For each item, state why it is on the list and what to do instead if the person thinks they need to touch it.

The “do not touch” list is not a sign of distrust. It is a way to shrink the decision space during an incident. When a person is paged at 2 a.m., they should not be wondering whether they are allowed to restart the database. The list answers that question in advance.

4. The First Production Action

Do not let the first production action be an emergency. Choose a low-risk, reversible task: read logs, view a dashboard, run a read-only query, or deploy a canary to a staging-like path. The person should perform the action while a more experienced teammate watches. This is not pair programming for its own sake; it is a controlled way to verify that the person’s credentials work, their mental model matches reality, and they can find the relevant runbook.

After the action, ask two questions: “What did you expect to happen?” and “What did you observe?” The gap between expectation and observation is where most early mistakes hide. If the person cannot explain the gap, they are not ready for unsupervised access.

5. Incident Response and the Recovery Checklist

Before access is granted, the person should know where the incident response runbook lives and how to start it. For this site’s audience, that means the recovery checklist is already written and tested. The conversation should walk through the first three steps of that checklist, not the whole document. The goal is to confirm the person can find the checklist, understand the severity levels, and know who to call when they are unsure.

Also cover the “stop and call” threshold. Define the conditions under which the person should stop making changes and escalate: unknown error messages, unexpected data loss, a security alert, or any action that affects a paying customer. The threshold should be low enough that a new operator feels safe pausing, but high enough that they do not escalate every routine warning.

6. Post-Action Review and the Access Log

Agree on how the team will review production actions. For a lean team, a lightweight post-action review works better than a formal postmortem for every change. The rule can be simple: any production action that triggers an alert, requires a rollback, or touches the “do not touch” list gets a 15-minute written review within 24 hours. The review should answer three questions: what happened, what surprised us, and what should change in the runbook or the access boundary.

Keep an access log. It does not need to be a dedicated tool; a shared document or a Git-tracked file works. Record the date, the person, the scope of access, the fallback contact, and the date of the next review. This log becomes the evidence trail when the team later asks, “Who had access to the database in March?” It also makes the access decision reversible and reviewable.

Person writing notes next to a laptop with code on screen

What the Conversation Should Produce

By the end of the session, the team should have a short written record. It does not need to be a formal policy document. A half-page note in the team wiki or a Git-tracked markdown file is enough. The record should include:

  • The person’s name and the date of first access.
  • The exact scope: which environments, which services, which IAM roles or SSH keys.
  • The named fallback contact for the first two weeks.
  • The “do not touch” list, with a one-line reason for each item.
  • The first production action the person will perform, and the expected result.
  • The review cadence: when the team will revisit the access decision.

This record is the team’s memory. It prevents the common failure mode where a person receives broad access during an emergency, the emergency ends, and the access quietly remains. A quarterly review of the access log catches that drift.

Common Failure Patterns and How the Conversation Prevents Them

Lean teams tend to make the same access mistakes. The pre-access conversation is a cheap way to interrupt each one.

Access Creep After an Incident

During an outage, a teammate may be given temporary admin access to unblock a deploy. After the outage, nobody revokes it. The pre-access conversation sets the expectation that temporary access has an expiration date and a named owner. The access log makes the expiration visible.

The “Read-Only” Illusion

Teams often say a new person has “read-only” access, but the actual IAM policy or SSH key allows more. The conversation should include a quick review of the exact policy document or key permissions. On AWS, that means reading the IAM policy JSON aloud. On GCP, it means checking the role bindings. On bare metal, it means checking sudoers. The person receiving access should be able to state what they can and cannot do in one sentence.

The Missing Fallback Contact

A new operator hits an unfamiliar error at 11 p.m. and does not know who to call. They either guess and make things worse, or they do nothing and the incident grows. The pre-access conversation names a specific fallback contact and sets the expectation that calling is not a failure. The fallback contact should be someone who has production experience and is willing to answer questions for the first two weeks.

The Untested Recovery Path

Access is granted, but the person has never seen the backup restore process. When the first real incident happens, they discover the restore takes four hours, not forty minutes. The pre-access conversation should include a walkthrough of the recovery checklist, not just a link to it. If the team has not tested the restore recently, that is a separate gap to close before granting access.

How to Adapt the Conversation for Different Team Sizes

The core topics stay the same, but the format changes with team size.

Two to five engineers: The conversation is informal but still written down. The person receiving access is often a founder or an early engineer who already has broad context. The risk is not ignorance but overconfidence. The conversation should focus on the “do not touch” list and the fallback contact, because the team is too small to absorb a long outage.

Six to fifteen engineers: The conversation becomes a short checklist that a team lead or senior engineer runs. The access log moves into a shared document or a Git repo. The review cadence becomes quarterly. The “do not touch” list is maintained by the team, not by one person.

Bare-metal or hybrid teams: The conversation adds a hardware layer. The person needs to know which physical hosts are production, how to reach the out-of-band management interface, and what happens if a reboot does not come back. The fallback contact should include someone who can physically access the hardware if needed.

What to Do After the Conversation

The conversation is not the end of the access decision. It is the start of a short probation period. For the first two weeks, the person’s production actions should be visible to the fallback contact. That visibility can be as simple as a shared Slack channel where the person posts a one-line note before and after each production change. The note does not need approval; it needs visibility.

At the end of the probation period, the team reviews the access log and asks three questions: Did the person follow the expected hygiene? Did any action surprise the team? Is the access scope still correct? If the answers are yes, yes, and yes, the access becomes routine. If not, the team adjusts the scope or extends the probation.

Team meeting around a table with laptops and notes

Frequently Asked Questions

How long should the first access conversation take?

Thirty minutes is usually enough for a focused session. The goal is not to cover every possible failure mode. It is to establish the production boundary, the “do not touch” list, the fallback contact, and the first low-risk action. If the environment is unusually complex, split the conversation into two sessions: one for the boundary and access scope, one for the recovery checklist and incident response.

Should we use a formal access request form?

For a lean team, a formal form is often overkill. A half-page note in the team wiki or a Git-tracked markdown file is enough. The key is that the record exists and is reviewed. If the team grows beyond fifteen engineers or enters a compliance-sensitive industry, a lightweight form with the same fields can replace the note without changing the underlying process.

What if the person needs access during an emergency before the conversation happens?

Grant the minimum access needed to resolve the emergency, and schedule the full conversation within 24 hours. The emergency access should be time-bound and named in the access log. After the conversation, the team should review whether the emergency access was broader than necessary and revoke or narrow it. This is a common pattern in lean teams, and the access log is what makes it visible.

How often should we review existing production access?

Quarterly is a reasonable cadence for a team of two to fifteen engineers. The review does not need to be a meeting. A single person can pull the access log, check the current IAM roles or SSH keys, and flag any drift. The review should answer one question: does every person with production access still need it at the current scope? If the answer is no, narrow or revoke.

Next Step: Pair the Conversation with a Tested Recovery Path

The first access conversation is only as strong as the team’s ability to recover from a mistake. If the person receiving access has never seen a restore work, the conversation is incomplete. The natural next step is to run a short recovery drill with the new person as the operator. That drill should follow the team’s existing recovery checklist and produce a written note about what worked and what did not. Over time, these notes become the team’s own evidence base for what production access should look like.

by ·Comments Off on How to Maintain a Service You Inherited from a Team That No Longer Exists

How to Maintain a Service You Inherited from a Team That No Longer Exists

Inheriting a production service from a team that no longer exists is a strange kind of ownership. You now hold the runtime, the deployment path, the access model, the alerting, and the recovery assumptions—without any of the original context. This is not a handoff. It is an archaeology project with a live site attached. For lean technical teams of two to fifteen engineers running cloud-native infrastructure on AWS, GCP, or bare metal, the danger is not that the service is complicated. The danger is that the service is silent about its own failure modes. The work is to make the inherited system legible, recoverable, and boring again.

This article walks through a repeatable sequence for taking over an orphaned service: inventory what actually runs, map the failure and recovery path, reduce access and configuration drift, rebuild monitoring around the metrics you can verify, and run a recovery drill before the first real incident. The goal is not to rewrite the service. The goal is to make the next engineer—possibly you at 02:00—able to operate it without tribal knowledge.

Two engineers reviewing a whiteboard diagram of a service architecture
Start by drawing the system as it actually runs, not as the old documentation claims it runs.

Start with a Runtime Inventory, Not the README

The first artifact to build is a runtime inventory: what is deployed, where it runs, what it depends on, and who can touch it. Do not start with the repository README. READMEs describe intent. The runtime describes behavior. In inherited systems, the two often diverge by months or years.

For AWS, begin with aws resourcegroupstaggingapi get-resources filtered by the service tag, then cross-check with aws ec2 describe-instances, aws rds describe-db-instances, and aws lambda list-functions. For GCP, use gcloud asset search-all-resources with a project or label filter. For bare metal, start with ss -tulpn on each host and compare the listening ports to the process list. The inventory should answer four questions per component: what is it, who deploys it, what does it call, and what calls it.

Record the inventory in a plain text file or a table in the repository. Do not build a new tool for this. A docs/ directory with one file per component is enough. The inventory is not documentation for its own sake. It is the input for the recovery checklist and the access review that follow.

Map the Failure and Recovery Path Before Touching Anything

Once the inventory exists, map the failure path for the two or three most likely incidents: database unavailable, dependency timeout, disk full, credential expiry, and deploy rollback. For each, write the exact commands or console steps required to detect the failure, stop the bleeding, and restore service. This is the same discipline as writing the recovery checklist before you need it, but applied to a system you did not build.

The recovery path should include the rollback command for the current deployment mechanism. If the service deploys via kubectl apply, the rollback is kubectl rollout undo deployment/<name>. If it deploys via AWS CodeDeploy, the rollback is an aws deploy rollback-deployment call or a console action. If it deploys via a shell script on a bare-metal host, the rollback is a copy of the previous binary or configuration. If you cannot identify the rollback path in under ten minutes, that is a finding, not a footnote. Write it down and schedule a fix.

Do not attempt to improve the architecture during this phase. The inherited service is in production. The first job is to make the current state recoverable. Improvements come after the recovery drill proves the baseline works.

Reduce Access and Configuration Drift in the First Week

Orphaned services accumulate access. Former team members retain IAM roles, SSH keys, API tokens, and database credentials. The first week of ownership should include an access review that removes every identity that cannot be tied to a current on-call or deployment need.

On AWS, use aws iam get-account-authorization-details and look for users and roles with unused access keys. The PasswordLastUsed and AccessKeyLastUsed fields are the evidence. On GCP, use gcloud projects get-iam-policy and compare members against the current team roster. On bare metal, review /etc/sudoers, ~/.ssh/authorized_keys, and any service account files. Remove or disable anything stale. If a credential is shared, rotate it and store the new value in the team’s existing secret manager—AWS Secrets Manager, GCP Secret Manager, or a self-hosted vault.

Configuration drift is the second target. Compare the running configuration to the repository. For Kubernetes, kubectl get deploy -o yaml against the committed manifests. For AWS, aws cloudformation describe-stacks or aws ssm get-parameter against the stored values. For bare metal, diff /etc/ files against the configuration management repo. Every difference is either a manual fix that was never committed or a configuration change that was never applied. Both are risks. Record them and close the gap.

A laptop showing a terminal with configuration files and a diff view
Configuration drift is easiest to see when you diff the running system against the repository.

Rebuild Monitoring Around Metrics You Can Verify

Inherited services often have alerting that is either too noisy or too quiet. The old team tuned it for their own mental model. You need to rebuild it around the four signals that matter for a service without dedicated SRE coverage: latency, errors, saturation, and a business-level health check.

Start with the metrics you can verify directly. For a web service, that means request duration at the load balancer or application, error rate from the application logs or the load balancer metrics, and a synthetic check that exercises a real user path. On AWS, CloudWatch metrics for ALB request count, target response time, and HTTP 5xx count are the baseline. On GCP, Cloud Monitoring metrics for the load balancer and the application. On bare metal, Prometheus with the node_exporter and an application exporter if one exists.

Do not import the old team’s dashboards wholesale. Rebuild them from the current runtime inventory. Each dashboard panel should map to a component in the inventory and a step in the recovery checklist. If a panel does not help you decide whether to page someone, remove it. Alert on symptoms, not on individual host metrics. A high CPU alert is only useful if it predicts a user-facing failure. A high error rate alert is useful immediately.

Run a Recovery Drill Before the First Real Incident

The recovery checklist is a hypothesis until you run it. Schedule a one-hour drill in the first two weeks of ownership. Pick the most likely failure from the failure path map—usually a database restart or a dependency timeout—and execute the checklist exactly as written. Do not skip steps. Do not improvise. The drill is a test of the checklist, not a test of your memory.

During the drill, record every place where the checklist was wrong, incomplete, or ambiguous. Did the rollback command require a flag that was not documented? Did the health check pass before the service was actually ready? Did the alert fire at all? Each gap is a work item. Fix the checklist immediately after the drill, then run the drill again the following week. Two successful drills in a row is the minimum bar for calling the service recoverable.

This is the same pattern used in Google’s SRE approach to emergency response, where the goal is to make the response path mechanical before the incident adds pressure. For a lean team, the drill is the only reliable way to build that muscle without a dedicated SRE.

Document the Service as a Set of Runbooks, Not a Wiki

The final step is to write the runbooks. A runbook is a short, command-first document for a specific operational task: deploy, rollback, restart, scale, credential rotation, and incident response. Each runbook should fit on one screen. If it does not, split it.

Store the runbooks in the same repository as the service, in a runbooks/ directory. This keeps the operational knowledge next to the code it operates. Use plain Markdown or plain text. The runbook should include the exact command, the expected output, and the rollback or verification step. Do not write paragraphs of explanation. Write the command, the check, and the undo.

For the incident response runbook, include the escalation path, the communication channel, and the link to the recovery checklist. The runbook is not a substitute for the checklist. It is the entry point that tells the on-call engineer where to start.

Tradeoffs and What to Skip

There are several tempting projects to skip during the first month of ownership. Do not rewrite the service. Do not migrate it to a new platform. Do not introduce a new observability stack. Do not refactor the deployment pipeline. Each of these projects consumes the time you need for the inventory, the access review, the monitoring rebuild, and the recovery drill. The inherited service is already in production. The risk is not that it is old. The risk is that it is unowned.

There is also a tradeoff in how much to automate. For a lean team, automation is valuable only when the underlying process is stable. Automating a broken recovery path just makes the failure faster. Run the manual checklist first. Automate only the steps that survive two successful drills.

What This Creates for the Site

This article is part of a recurring column on taking over orphaned infrastructure. The next article in the series will cover how to run a service handoff interview when the previous team is still partially available, including the ten questions that produce the most useful operational answers. If you have inherited a service and found a gap in this sequence, send a note through the contact page. Reader questions become the next runbook.

A small team gathered around a monitor during an incident review
The goal is to make the next incident boring: detected, diagnosed, and recovered from a checklist.

Frequently Asked Questions

What is the first thing to do when you inherit a service from a team that no longer exists?

Build a runtime inventory. List every running component, its dependencies, and its access model. Use cloud provider APIs or host-level commands to capture what actually runs, not what the old documentation claims. The inventory is the input for the recovery checklist, the access review, and the monitoring rebuild.

How do you know if an inherited service is recoverable?

Run a recovery drill. Pick the most likely failure, execute the recovery checklist exactly as written, and record every gap. Fix the checklist and run the drill again. Two successful drills in a row is the minimum bar for calling the service recoverable. A checklist that has never been run is a guess.

Should you rewrite an inherited service to make it easier to maintain?

Not in the first month. The service is already in production. The immediate risk is that it is unowned, not that it is old. Focus on inventory, access review, monitoring, and recovery drills. A rewrite or platform migration can be evaluated after the service is operationally legible and recoverable.

What is the difference between a runbook and a recovery checklist?

A runbook is a command-first document for a specific operational task, such as deploy, rollback, or credential rotation. A recovery checklist is the ordered sequence of steps for responding to a specific failure. The runbook is the entry point. The checklist is the response path. Both should be stored in the service repository.

by ·Comments Off on What Changes in Your Backups When You Start Complying with Retention Regulations

What Changes in Your Backups When You Start Complying with Retention Regulations

Retention regulations turn a backup system from a recovery tool into a records system. For a lean technical team running cloud-native infrastructure on AWS, GCP, or bare metal, that shift changes the backup schedule, the storage tier, the deletion policy, the restore test, and the access model. The main entity here is the regulated retention schedule: a documented, enforceable rule that says which backup artifacts must exist, for how long, in what form, and who may delete them. Adjacent concepts include legal hold, immutable storage, data classification, retention classes, disposition, and audit evidence. This matters because a team of two to fifteen engineers rarely has a compliance officer. The person who writes the backup script is often the same person who answers the auditor’s question. If the backup design does not encode retention rules as operational controls, the team will discover the gap during an incident or an audit, not during a design review.

Server racks in a data center with blue and white cabling

Retention Regulations Change the Default from “Keep Until Space Runs Out” to “Keep Until the Rule Says Delete”

Most small infrastructure teams start with a simple retention model: keep daily snapshots for seven days, weekly snapshots for four weeks, monthly snapshots for six months, then let the storage lifecycle delete the rest. That model optimizes for recovery and cost. A retention regulation adds a second axis: the minimum retention period for a class of data. For example, a payment processor may need to retain transaction backups for 24 months under a card network rule, while a healthcare-adjacent service may need six years for certain records under a national health data rule. The operational change is not just “keep longer.” It is that deletion becomes a controlled action with a documented justification, and the backup catalog must prove that the artifact existed for the full required window.

On AWS, this often means moving from a single S3 lifecycle rule to a combination of S3 Object Lock in governance or compliance mode, AWS Backup vault locks, and separate vaults per retention class. On GCP, the equivalent is a bucket with a retention policy and a retentionPolicy.retentionPeriod set in seconds, plus a separate bucket for data that must not be locked. On bare metal, the same logic applies with ZFS snapshots and a separate immutable dataset, or with BorgBackup repositories that are mounted read-only after the retention window closes. The common pattern is not a specific vendor feature. It is the separation of recoverable backups from retained records.

First Change: You Stop Treating All Backups as One Class

Before retention regulations, a lean team can often get away with one backup policy for everything: the database, the object store, the configuration files, and the container images all follow the same schedule. Once a regulation applies, that single policy becomes a liability. A 30-day snapshot policy for a test database is fine, but the same policy applied to a regulated production database creates a compliance gap. The first operational change is to classify data by retention requirement and map each class to a named backup vault, bucket, or repository.

A practical classification for a small team is three tiers:

  • Operational backups: short retention, frequent restore tests, no legal hold. These are the normal snapshots and incremental backups used for recovery from failed deployments or bad migrations.
  • Regulated records: long retention, immutable storage, deletion only after a documented disposition review. These are the backups that an auditor may request.
  • Excluded data: ephemeral caches, build artifacts, and test data that should never enter a regulated vault because they create noise and increase storage cost.

This classification is not a one-time spreadsheet. It becomes a field in the backup job definition. In AWS Backup, that means separate backup plans per resource tag. In GCP, it means separate bucket names with a suffix like -retained or -operational. On bare metal, it means separate ZFS datasets or Borg repositories. The goal is that an engineer can look at a backup job and know which retention class it belongs to without opening a policy document.

Second Change: Deletion Becomes a Controlled Operation

The most visible change when retention regulations apply is that you can no longer delete a backup just because it is old or because a cleanup script found it. Deletion becomes a two-step process: first, confirm that the retention period has expired; second, record the deletion in a log that can be shown to an auditor. On AWS, S3 Object Lock compliance mode prevents deletion even by the root account until the retention date passes. On GCP, a bucket retention policy does the same. On bare metal, ZFS zfs hold or a read-only Borg repository provides a similar control, though it requires more discipline because there is no cloud provider enforcing the lock.

The operational shift is that the backup cleanup script changes from a simple find -mtime +30 -delete to a script that checks a retention manifest before deleting anything. The manifest can be a JSON file in the repository, a database table, or a set of tags on the backup objects. The key property is that the manifest is written at backup time, not at deletion time. If the manifest is written at deletion time, an engineer can accidentally delete a backup that should have been retained because the manifest did not exist yet. A simple pattern is to write a retention.json file into each backup directory with the creation date, the retention class, and the earliest deletion date. The cleanup script reads that file and refuses to delete anything before the earliest deletion date.

Person writing on a clipboard next to a laptop and server equipment

Third Change: Restore Tests Must Cover the Regulated Retention Window

A backup that cannot be restored is not a backup. That principle does not change under retention regulations, but the test scope does. Before regulations, a lean team might test the most recent daily backup once a month and call it done. After regulations, the team must prove that a backup from any point in the retention window can be restored. That means testing a 12-month-old backup, a 24-month-old backup, and a backup that is one day before its deletion date. The reason is simple: a backup that was written correctly 18 months ago may not restore today because the restore tool changed, the encryption key rotated, or the database version drifted.

A repeatable pattern is to schedule a quarterly restore drill that picks a random date from the regulated retention window and restores that backup to a staging environment. The drill should be documented in the same runbook as the normal recovery test. The recovery checklist should include a step for verifying the retention metadata on the restored artifact, not just the data itself. If the restore succeeds but the retention metadata is missing, the backup is not compliant even though it is technically recoverable.

On AWS, this drill can use AWS Backup restore jobs to a staging VPC. On GCP, it can use a separate project with a restore test service account. On bare metal, it can use a spare machine or a container that mounts the backup repository read-only. The common requirement is that the restore test is automated enough to run without a senior engineer manually driving every step, but manual enough that a human verifies the data is actually usable.

Fourth Change: Access to Backups Becomes a Security Boundary, Not Just an Operational Convenience

Before retention regulations, backup access is often broad. Any engineer who can deploy code can probably also list and restore backups. That is convenient for debugging and for recovering from a bad deploy. Once backups become regulated records, that broad access becomes a risk. An engineer who can delete a backup can destroy evidence. An engineer who can restore a backup can exfiltrate regulated data without touching the production database. The operational change is to separate the backup operator role from the backup auditor role and from the backup restorer role.

On AWS, this means using IAM policies that allow backup:StartRestoreJob only for a specific role, and s3:DeleteObject only for a role that is not used by day-to-day operations. On GCP, it means using a dedicated service account for backup deletion and a separate service account for restore. On bare metal, it means using SSH keys or filesystem permissions that separate the backup user from the restore user. The goal is not to make access impossible. It is to make access auditable. Every restore and every deletion should produce a log entry that includes the actor, the backup ID, the retention class, and the timestamp.

This change also affects the monitoring and alerting stack. A lean team that lacks dedicated SRE coverage often monitors backup success but not backup access. Under retention regulations, the team should alert on any deletion attempt that is blocked by an immutable lock, any restore of a regulated backup, and any change to the retention policy itself. These alerts are not about preventing an incident. They are about detecting a compliance-relevant action early enough to investigate it.

Fifth Change: The Backup Catalog Becomes an Audit Artifact

Before regulations, a backup catalog is a convenience. It tells you what backups exist and where they are. After regulations, the catalog becomes evidence. An auditor may ask for a list of all backups of a specific database for the past 24 months, with proof that each backup existed for the full retention period. If the catalog is a spreadsheet that someone updates by hand, that request will be painful. If the catalog is generated from the backup system’s API, it is a query.

The operational change is to treat the backup catalog as a first-class artifact. On AWS, that means enabling AWS Backup audit reports and exporting them to a locked S3 bucket. On GCP, it means using Cloud Audit Logs for bucket operations and exporting them to a separate project. On bare metal, it means writing a small script that lists all snapshots or Borg archives and writes the output to a timestamped file in a read-only location. The catalog should include at least these fields: backup ID, source resource, retention class, creation time, earliest deletion time, storage location, and encryption key ID.

This catalog also becomes the input for the deletion script. Instead of deleting based on file age, the script deletes based on the catalog’s earliest_deletion_time field. That single change removes the most common cause of accidental deletion: a cleanup script that uses the wrong clock or the wrong timezone.

Sixth Change: Encryption Key Management Gets More Deliberate

Retention regulations often require that retained backups be encrypted, but the operational change is not just enabling encryption. It is managing the keys so that a backup from 18 months ago can still be decrypted today. A common failure pattern is to rotate the KMS key or the GPG key and then discover that old backups are unreadable because the old key was destroyed. Under retention regulations, key destruction becomes a compliance event, not just a security hygiene task.

On AWS, this means using a separate KMS key for regulated backup vaults and disabling automatic key rotation if the rotation would break old backups. On GCP, it means using a Cloud KMS key with a rotation period that is longer than the longest retention window, or using a key version that is never destroyed. On bare metal, it means storing the GPG or age encryption keys in a hardware token or a password manager with a documented recovery procedure. The key management policy should answer one question: if the only person who knows the passphrase leaves the company, can the team still restore a 24-month-old backup?

Close-up of a server rack with glowing blue lights and network cables

What Does Not Change

Retention regulations do not change the fundamental backup principles. You still need to test restores. You still need to monitor backup success. You still need to document the recovery procedure. You still need to keep the backup system simple enough that a tired engineer can operate it at 3 a.m. The regulations add constraints, but they do not remove the need for operational discipline. A team that has no backup testing before regulations will not become compliant just by enabling S3 Object Lock. The lock prevents deletion, but it does not prevent a restore failure caused by a missing dependency or a changed schema.

What changes is the default posture. Before regulations, the default is to delete old backups to save money. After regulations, the default is to retain backups until a documented rule says they can be deleted. That shift is small in code but large in culture. It means the team stops thinking of backups as a cost center and starts thinking of them as a records system with a retention schedule, an access policy, and an audit trail.

Practical Starting Point for a Lean Team

If you are starting from a single backup policy and need to add retention compliance, do not rebuild everything at once. Start with three steps:

  1. Write down the retention classes in a single page. Name the class, the retention period, the storage location, and the deletion rule. If you cannot write it in one page, you do not understand it yet.
  2. Move the regulated data into a separate vault or bucket with immutable retention enabled. Do not try to retrofit immutability onto the existing operational backups. A separate location is easier to reason about and easier to audit.
  3. Add a deletion manifest to the backup job. Write a small JSON file with the retention metadata at backup time. Change the cleanup script to read that manifest and refuse to delete anything before the earliest deletion date.

These three steps do not require a new tool or a new vendor. They require a change in how the team thinks about backups. The rest — restore drills, access separation, catalog exports, key management — can follow incrementally. The important thing is that the first step creates a clear boundary between operational backups and regulated records. Once that boundary exists, every other decision becomes easier.

FAQ

Do retention regulations require me to keep every backup forever?

No. Retention regulations set a minimum retention period for specific classes of data. Once that period expires, you can delete the backup if no legal hold applies. The operational change is that deletion must be documented and must not happen before the minimum period. A common mistake is to confuse the backup retention policy with the data retention policy. A database may need to retain transaction records for seven years, but that does not mean every daily backup must be kept for seven years. The backup retention period is usually shorter than the data retention period because the live database itself is the primary record.

What is the difference between governance mode and compliance mode in S3 Object Lock?

Governance mode allows users with a specific IAM permission to delete objects before the retention date. Compliance mode prevents deletion by any user, including the root account, until the retention date passes. For regulated backups, compliance mode is the safer default because it removes the risk of an accidental or malicious deletion by an engineer with broad permissions. The tradeoff is that compliance mode makes it harder to fix a mistake, such as a backup that was written to the wrong bucket. A lean team should use compliance mode for the regulated vault and governance mode for the operational vault.

How do I prove to an auditor that a backup existed for the full retention period?

The proof is the backup catalog plus the storage system’s audit log. The catalog shows the backup ID, creation time, and retention class. The audit log shows that no deletion occurred before the retention date. On AWS, AWS Backup audit reports and S3 server access logs provide this evidence. On GCP, Cloud Audit Logs for bucket operations provide it. On bare metal, a timestamped catalog file in a read-only location plus filesystem or ZFS snapshot metadata provides it. The key is that the catalog is generated automatically from the backup system, not maintained by hand.

by ·Comments Off on How to Evaluate Whether a New Tool Reduces or Transfers Your Operational Risk

Operational risk is the chance that a failure in people, processes, or systems will degrade a service you own. For a lean technical team running cloud-native infrastructure on AWS, GCP, or bare metal, a new tool is rarely neutral. It either reduces the probability or blast radius of an incident, or it moves that risk somewhere else: into a vendor relationship, a configuration surface, a credential boundary, or an on-call workflow that nobody fully owns. This article gives you a repeatable way to tell the difference before you commit.

We will use a simple frame: risk reduction means the tool removes a failure mode you can name and measure. Risk transfer means the tool changes where the failure will appear, who will feel it first, and how long it will take to recover. Both can be acceptable. What is not acceptable is adopting a tool because it feels safer while leaving the actual failure path unexamined.

Two engineers reviewing a whiteboard with a system diagram and risk notes
Start with the failure path, not the feature list.

Define the Failure Mode Before You Evaluate the Tool

Most tool evaluations begin with a demo and end with a procurement form. That order hides the most important question: which specific failure are you trying to make less likely or less expensive?

Write the failure mode as a sentence. For example:

  • “A bad deploy to the primary PostgreSQL instance leaves us with 40 minutes of downtime because the last tested restore is from the previous night.”
  • “A leaked AWS access key in a public repository gives an attacker write access to the production S3 bucket for up to 12 hours because we only review IAM activity weekly.”
  • “A noisy neighbor on a shared bare-metal host causes latency spikes in the checkout service, and we cannot prove which workload is responsible.”

If you cannot write the failure mode, you are not evaluating a tool. You are evaluating a feeling. That feeling is usually “the vendor’s dashboard looks more complete than ours.”

Separate the Four Risk Surfaces a Tool Touches

Every operational tool touches at least one of four surfaces. A tool that reduces risk on one surface can quietly increase it on another.

1. The failure path itself

This is the direct effect. A backup tool that performs nightly restores to a staging environment reduces the risk of discovering a broken backup during an incident. A monitoring agent that exports per-process memory pressure on bare-metal hosts reduces the time to identify a noisy neighbor. These are direct reductions.

2. The credential and access boundary

Many tools require broad read access to your cloud provider, your database, or your container runtime. A monitoring SaaS that asks for ReadOnlyAccess across all AWS accounts is not just an observer; it is a new credential boundary. If that vendor is compromised, or if an API key leaks from a CI pipeline, the tool becomes an attack surface.

Ask: does this tool need cross-account access, or can it run with per-service IAM roles and a narrow policy? A tool that can operate with scoped roles and short-lived credentials transfers less risk than one that wants a static key with organization-wide read access.

3. The recovery workflow

A tool can reduce the chance of a failure while making recovery slower. For example, a database proxy that adds connection pooling and automatic failover may also add a new component that must be running before the application can reconnect. If the proxy itself fails, the recovery path now includes “restart the proxy, verify its configuration, and check its quorum.” That is a new step in the middle of an incident.

Before adopting a tool, write the recovery steps for the failure mode with and without the tool. If the tool adds more than one new step to the recovery path, treat that as a cost, not a feature.

4. The on-call and training surface

A tool that only one engineer understands is a single point of failure in human form. If that engineer leaves, the team inherits a black box. The risk has not disappeared; it has moved into the team’s memory and documentation.

For a team of two to fifteen engineers, the rule should be simple: no tool enters production unless at least two people can explain its failure modes and its recovery steps. If you cannot meet that bar, the tool is a risk transfer to your own bus factor.

A small team gathered around a laptop reviewing an incident timeline
If only one person can recover a tool, the tool owns you during an incident.

Use a Five-Question Scorecard

You do not need a weighted matrix with twenty criteria. Five questions will catch most bad decisions.

Question 1: Which named failure mode does this tool reduce?

If the answer is “general reliability” or “better visibility,” stop. Those are not failure modes. They are marketing categories. A legitimate answer names a specific incident pattern: failed restores, silent disk exhaustion, expired TLS certificates, orphaned EBS volumes, or credential sprawl.

Question 2: What new failure mode does this tool introduce?

Every tool has at least one. A centralized secrets manager introduces the failure mode “the secrets manager is unreachable, so no service can start.” A Kubernetes operator introduces the failure mode “the operator’s reconciliation loop is stuck, so the desired state is not applied.” A log aggregation SaaS introduces the failure mode “the log pipeline is down, so we are blind during an incident.”

Write the new failure mode in the same sentence format as the original one. If you cannot, you have not looked hard enough.

Question 3: Does the tool reduce the probability, the blast radius, or the time to detect?

These are different. A tool that reduces probability makes the failure less likely. A tool that reduces blast radius makes the failure less expensive when it happens. A tool that reduces time to detect makes the failure visible sooner.

For example, automated certificate renewal reduces probability. Read-only database replicas reduce blast radius for reporting queries. Synthetic checks on the checkout endpoint reduce time to detect. A tool that claims to do all three is usually doing none of them well.

Question 4: What is the recovery path when the tool itself fails?

This is the question most teams skip. If the tool is a SaaS monitoring platform, what happens when the platform is down? Do you lose alerting entirely, or do you have a local fallback? If the tool is a database proxy, what happens when the proxy crashes? Can the application bypass it, or is the proxy in the critical path?

A good answer includes a tested fallback. A bad answer is “we will figure it out during the incident.”

Question 5: Can we run a one-hour drill that proves the tool works under failure?

If the tool cannot be tested in a controlled drill, it is not operational. It is aspirational. For backup tools, the drill is a restore to a clean environment. For monitoring tools, the drill is killing a service and confirming the alert fires within the expected window. For access tools, the drill is revoking a credential and confirming the service fails closed.

This is where the recovery checklist becomes useful. Write the checklist before you need it, and run it against the tool before you sign the contract.

Three Common Risk Transfers That Look Like Risk Reductions

1. The “single pane of glass” monitoring platform

A unified dashboard that aggregates metrics, logs, and traces feels like a reduction in cognitive load. But it often transfers risk into a single vendor’s availability and a single team’s configuration. If the platform is down, you lose all three signals at once. If the platform’s query language is proprietary, you lose the ability to move quickly when pricing or terms change.

A leaner approach is to keep raw metrics in a system you control, such as Prometheus on your own infrastructure, and use the SaaS only for long-term storage or alert routing. That way, a vendor outage degrades your visibility instead of eliminating it.

2. The “zero-configuration” database service

Managed database services reduce the operational burden of patching, replication, and failover. But they transfer risk to the provider’s backup schedule, restore SLA, and network path. If the provider’s restore process takes six hours and your recovery target is one hour, the tool has not reduced your risk; it has moved it into a contract you did not read closely enough.

Before adopting a managed service, ask for the actual restore time from a recent incident, not the marketing SLA. Then run your own restore drill. The gap between the two numbers is the risk you are accepting.

3. The “AI-powered” anomaly detector

Anomaly detection tools promise to find problems before they become incidents. In practice, they often generate a stream of low-signal alerts that on-call engineers learn to ignore. The risk has not been reduced; it has been transferred into alert fatigue. The failure mode is now “the real alert was buried under forty anomaly notifications, and nobody looked at it for three hours.”

If you adopt an anomaly detector, pair it with a strict alert budget. If the tool cannot stay within the budget during a two-week trial, it is adding noise, not signal.

A laptop screen showing a dense alert dashboard with multiple red indicators
Alert fatigue is a risk transfer, not a risk reduction.

Build a Pre-Adoption Drill

The best way to evaluate a tool is to run a small, time-boxed drill before you commit. The drill should take no more than one hour and should answer three questions:

  1. Does the tool detect the failure mode you named?
  2. Does the tool’s own failure create a new incident?
  3. Can a second engineer recover the tool without calling the vendor?

For a backup tool, the drill is: delete a non-critical table, restore it from the tool, and time the result. For a monitoring tool, the drill is: stop a service, confirm the alert fires, then kill the monitoring agent and confirm you still have a fallback signal. For an access tool, the drill is: revoke a credential, confirm the service fails closed, then restore access and confirm the service recovers.

If the tool cannot pass a one-hour drill, it will not pass a 3 a.m. incident.

Document the Decision as a Risk Ledger Entry

Every tool adoption should produce a short entry in a risk ledger. The entry does not need to be long. Four lines are enough:

  • Failure mode reduced: nightly restores were untested; now restored weekly to staging.
  • New failure mode introduced: backup agent can consume up to 30% CPU during the backup window.
  • Recovery path if the tool fails: disable the agent, fall back to nightly pg_dump until the agent is fixed.
  • Owner: two engineers, with a runbook in the team wiki.

This ledger becomes the team’s institutional memory. When someone asks “why did we choose this tool?” the answer is not a vendor whitepaper. It is a record of the failure mode, the tradeoff, and the tested fallback.

When a Risk Transfer Is the Right Call

Risk transfer is not always bad. A two-person team should not run its own Kafka cluster if a managed alternative exists. The key is to make the transfer explicit and to price it correctly.

A managed service transfers operational risk to the vendor, but it also transfers control. You accept the vendor’s backup schedule, the vendor’s restore time, and the vendor’s incident communication. If those are acceptable, the transfer is rational. If they are not, you have not reduced risk; you have hidden it.

The test is simple: can you name the new failure mode and its recovery path? If yes, the transfer is a decision. If no, the transfer is a hope.

FAQ

What is the difference between reducing risk and transferring risk?

Reducing risk means a tool makes a specific failure mode less likely, less expensive, or faster to detect. Transferring risk means the tool moves the failure somewhere else: into a vendor’s availability, a new credential boundary, a new component in the critical path, or a single engineer’s knowledge. A transfer can be acceptable, but only if you can name the new failure mode and its recovery path.

How do we evaluate a tool when we do not have a dedicated SRE team?

Use the five-question scorecard and the one-hour drill. The scorecard forces you to name the failure mode, the new failure mode, the type of reduction, the recovery path, and the testability. The drill proves the tool works under failure. If a tool cannot pass both, it is not appropriate for a lean team, regardless of what the vendor’s marketing says.

What is the most common risk transfer teams miss?

Alert fatigue from anomaly detection tools. The tool appears to reduce risk by finding problems early, but in practice it generates a stream of low-signal alerts that on-call engineers learn to ignore. The real incident is then buried under noise. The fix is to set an alert budget and reject any tool that cannot stay within it during a two-week trial.

Should we avoid all managed services because they transfer risk?

No. Managed services are often the right choice for a small team. The point is to make the transfer explicit. Read the restore SLA, run your own restore drill, and document the new failure mode. If the managed service’s failure mode is acceptable and recoverable, the transfer is a rational decision, not a hidden risk.

This article is part of a series on operational risk for lean technical teams. The next step is to write the recovery checklist for your most critical service before you evaluate any new tool. That checklist is the baseline against which every tool should be measured.