Every team that runs production long enough collects failures that keep coming back: a nightly restic job that stalls on a locked index, a PostgreSQL connection pool that saturates under one specific report, a pgBackRest archive gap that shows up after weekend batch work. Recurring-failure triage — the decision to spend engineering hours on a code fix or to keep the incident behind an operational workaround such as a runbook step, a cron script, or a widened alert threshold — is a routine call that quietly shapes reliability on a small team. The adjacent concepts matter as much as the verdict itself: error budgets, MTTR trends, workaround debt, blameless postmortems, and decision records. For a team of two to fifteen engineers with no dedicated SRE function, engineering hours are the scarcest resource in the system, and every workaround is a standing tax on on-call attention. This is the method we use. Thresholds included.

What counts as a recurring failure

Answer first: three occurrences of the same root cause inside a rolling ninety days is the working definition of recurring. Symptom clustering is not evidence. Three pages tagged latency can be three different diseases with three different correct responses, and treating them as one recurring failure is how a team ends up fixing the wrong thing twice.

Symptoms lie. A six-person fintech we will call Corvid kept paging on 5xx spike; incident review showed the three pages in one month were a CDN misconfiguration, a connection pool exhausted by a reporting query, and a slow migration holding locks. Three causes, three decisions — one vendor ticket, one code fix, one scheduling change. Tagging by root cause, not by alert name, is what makes the count mean anything.

The comparison step is a blameless postmortem exercise — the discipline Google’s SRE practice formalized in the SRE Workbook chapter on postmortem culture. On a small team the postmortem can be thirty minutes and a shared doc. The point is to compare causes, not to produce ceremony.

Below three occurrences, document and wait. Two is a coincidence worth a line in the incident record. One is just an incident. At three, the decision goes on the ledger — fix, workaround, or monitor — whether or not the answer feels obvious yet.

The decision frame: carrying cost versus fix cost

Promote a workaround to a code fix when the workaround’s carrying cost compounds faster than the fix’s one-time cost. Everything else in this method is detail.

Carrying cost has four line items. Frequency times minutes per occurrence times people involved. Pager load, because every page spends on-call attention even when the fix is easy. Execution-error risk, because manual steps have a failure rate and it is highest at 3 a.m. And drift, because workarounds rot as the system changes around them. Fix cost has three: engineer-hours, regression risk, and deploy risk on whatever path the change lands.

Concrete numbers make the tradeoff visible. A workaround firing every eleven days with a forty-minute MTTR and two people involved costs roughly 3.5 hours a month before you count the pager. A fix estimated at twelve engineer-hours pays for itself in under four months if it removes the failure outright, and in eight if it only halves the frequency. If the same fix has to land on the payment authorization path, the regression risk may dominate both numbers — which is why the frame has two sides.

When the code fix wins

The failure touches a recovery path

Any known defect in backup, restore, or WAL archiving gets a fix, not a workaround. Recovery paths get exercised at the exact moment the system is already down; a workaround there is a bet placed during your worst hour. restic check reporting index inconsistencies, a pgBackRest archive stall on PostgreSQL 16, a WAL gap discovered mid-restore — these are promotion cases with no counting period. PostgreSQL’s durability model depends on write-ahead log replay, and the WAL chapter of the PostgreSQL documentation is blunt about the ordering guarantees you are relying on. A workaround that leaves those guarantees unverified is not a workaround. It is an untested restore.

Detection is cheap. A monthly restore lottery — restore a randomly selected backup to a scratch host and time the result — catches workarounds that quietly rotted. If a workaround touches restore steps at all, those steps belong on a written recovery checklist, not in tribal memory; the approach is covered in Write the Recovery Checklist Before You Need It.

The workaround depends on someone’s memory at 3 a.m.

If the workaround’s steps live in one person’s head, that is a fix signal, not a runbook. The test is escalation behavior: when the PagerDuty escalation for this incident converges on a specific engineer — only Dana knows the restart order — the workaround is a single point of failure wearing a runbook costume. Promote it to code, or at minimum to a tested runbook step behind a synthetic check. Steps that require judgment under fatigue have an error rate. We have watched a correct six-step procedure get executed wrong twice in one quarter because step four depended on reading a stale Grafana dashboard.

Frequency and impact cross a threshold you wrote down in advance

Pick the thresholds in advance and write them down; do not negotiate them per incident. The set we use in these field notes: same root cause three times in ninety days with MTTR over thirty minutes — fix. Any occurrence with data-loss exposure — fix regardless of count. Frequency rising two months in a row — fix review, even if MTTR is short. Below those lines, a documented workaround is a defensible answer.

When the workaround wins

You do not own the failing layer

If the defect sits inside a managed service, the workaround is often the only lever you actually hold. Cloud SQL failover quirks, intermittent S3 slow-down responses during heavy list operations, a GCP load balancer behavior that only appears under one traffic shape — file the vendor case, keep the runbook step, and record the case ID in the ledger. The tradeoff: waiting on a vendor is a decision, not a default, so it gets a revisit date like any other workaround.

The fix’s blast radius exceeds the failure’s cost

Compare the fix’s regression risk against the failure’s measured cost, not against its annoyance. A four-person payments team we will call Heron kept a manual failover runbook for a flaky connection pool rather than touching the transaction path mid-quarter; the fix shipped later, in a planned window, with a rollback plan and a canary. The classic version of this branch is the PostgreSQL major-version upgrade that would remove the failure class entirely. The workaround is legitimate there — as a bridge with an end date, not as a residence.

The failure is bounded and the error budget is not

A bounded failure that stays inside its error budget is a candidate for a documented workaround. Bounded means capped blast radius, self-limiting duration, and no data exposure. If the monthly burn-rate alerts in Grafana still finish the month in the black, the workaround is defensible; the moment burn alerts fire two months running, the promotion review starts. This is the one branch where doing nothing expensive is often the correct answer — provided the ledger entry exists.

Write the decision down: the workaround ledger

Every accepted workaround gets a one-page decision record; undocumented workarounds are just failures you have agreed to forget. The format borrows from architecture decision records — context, decision, consequences — trimmed to what an on-call engineer will actually read:

• Incident IDs (PagerDuty references) that triggered the decision
• Root cause in one sentence
• Workaround steps, in runbook form
• Carrying-cost estimate: frequency, MTTR, people involved
• Promotion trigger — the threshold that would force a fix
• Revisit date and owner

The ledger review is a fixed thirty-minute quarterly slot with three questions: which entries fired since last review, which missed their revisit date, and which thresholds should change. Entries that miss their revisit date get promoted or retired. The ledger is allowed to shrink, and most quarters it should.

A worked example from the field

A nine-person B2B SaaS team on AWS — anonymized here as Kiln — ran restic 0.16.4 against a 4 TB repository, and the nightly prune failed roughly every eleven days with an index inconsistency (pack file cannot be found) that on-call resolved with restic rebuild-index and a retry. MTTR was about forty minutes, two people were usually involved, and the window produced three pages in ninety days.

The ledger entry made the arithmetic plain: about 3.5 hours a month of on-call attention, on a recovery path, crossing the three-in-ninety-days line. The fix — moving prune to a weekly systemd timer with a restic check pass beforehand, plus explicit lock cleanup — was estimated at ten engineer-hours and paid for itself inside a quarter. It shipped; occurrences went to zero and stayed there for two quarters. The follow-up mattered as much as the fix: the next restore lottery caught a stale copy of the old rebuild-index runbook step still linked from the wiki, and deleting it removed exactly the memory-dependent failure this method is designed to catch.

The counterfactual is worth stating. Had the same failure lived on the vendor’s side of the fence — say, inside a managed database’s snapshot scheduler — the same ledger entry would have justified keeping the workaround, with a case number and a revisit date instead of a sprint ticket. The method does not bias toward fixes. It biases toward deciding once, with numbers, on paper.

Frequently asked questions

How many recurrences justify a code fix?

Three occurrences of the same root cause in a rolling ninety days is a reasonable promotion trigger, provided MTTR exceeds about thirty minutes or the workaround needs more than one person. Any recurrence that touches backup, restore, or WAL archiving skips the count entirely and goes straight to a fix, because a workaround on a recovery path is a bet placed during your worst hour.

Is a runbook step enough, or is that just a workaround with better documentation?

A runbook step is the floor, not the finish line. It removes the memory dependency but none of the carrying cost — the pages, the minutes, the execution risk. If the step still fires monthly, it remains a promotion candidate. Runbooks are where workarounds wait; they are not where workarounds retire.

How do you track workaround debt across quarters?

One ledger file, one page per workaround: incident IDs, root cause, steps, carrying-cost estimate, promotion trigger, revisit date, owner. Review in a fixed thirty-minute quarterly slot. Any entry that misses its revisit date gets promoted or retired. The ledger should shrink most quarters; if it only grows, your promotion thresholds are too patient.

When is a monitoring change enough — no fix and no workaround?

When the failure is genuinely rare, self-limiting, and cheap to confirm, a synthetic check or an adjusted alert threshold can be the entire response. The test: if the check fired next month, would anyone change behavior? If no, you are collecting noise. If yes, you have a workaround by another name, and it belongs on the ledger.

Who makes the promotion call on a small team?

The engineer who carries the pager for that service proposes; the weekly ops slot confirms, with one other engineer as a sanity check. That avoids both failure modes of small-team decisions — the solo hero fix at 2 a.m., and the silent endurance of a workaround nobody officially accepted.

Where this goes next

The promotion decision and the recovery checklist are the same muscle: both are about writing down, in advance, what you will do when the system is at its worst. The companion piece is Write the Recovery Checklist Before You Need It, which covers the restore-path side of this method. And if you are holding a workaround you cannot decide about, send the ledger entry — anonymized recurring failures are the raw material of this column.