Every recurring failure eventually forces the same decision: change the code, or change how you operate around it. A recurring failure is a failure mode that has returned after you believed it was resolved — the third 3 a.m. page for the same PostgreSQL connection exhaustion, the second month with WAL archiving stalls, the weekly restic job that dies on a stale lock. The fix-or-workaround decision is the judgment call about whether the next occurrence gets a merged pull request or a runbook entry, and it touches everything else a small team manages: MTBF and MTTR tracking, error budgets, workaround debt, blameless postmortems, and runbook maintenance. For a team of two to fifteen engineers running production without a dedicated SRE function, this decision arrives at least monthly, usually under time pressure and from memory. This article replaces memory with a procedure: recurrence thresholds, a cost line you can compute in ten minutes, three anonymized incidents with three different verdicts, and a register format that keeps workarounds from rotting.

The short answer

A recurring failure deserves a code fix when three conditions hold at once. It has recurred at least three times in ninety days, or it has breached a stated guarantee such as your RPO. The fix has a bounded scope you can estimate in engineer-hours. And the recurring cost of living with the failure — pages, manual recovery, workaround maintenance — exceeds the one-time cost of the fix within roughly two quarters. It deserves an operational workaround when any one of those conditions fails: the root cause sits in code you do not own, the fix is unbounded, or the workaround is cheap, durable, and reversible.

Both paths are legitimate. The failure mode to avoid is the unmade decision — the workaround nobody agreed to, written nowhere, reviewed never, quietly becoming permanent infrastructure. Everything below is about gathering evidence for those three conditions and keeping the workaround path honest.

Count recurrences before you debate solutions

Arguments about fixes are usually arguments about frequency conducted from memory. Replace the memory with a count. Export the last ninety days of incidents from PagerDuty or your paging tool, tag everything belonging to the same failure mode — one tag such as db-conn-exhaust is enough — and count. If you have never tagged incidents, start with the failure you already suspect; you will likely find more occurrences than anyone remembered.

Weight the count by impact. Three warnings logged during business hours are not equivalent to three pages at 3 a.m. Google’s SRE book treats unreliability as an error budget — a fixed quantity you may spend per quarter — and the metaphor works even without formal SLOs: each recurrence spends attention you do not get back, and a tag that has paged the on-call three times in a quarter has spent enough.

Record MTBF (mean time between failures) and MTTR (mean time to restore) from the incident timestamps. One reader team of four tracked a nightly restic lock failure for six weeks before deciding anything: MTBF 1.6 days, MTTR eleven minutes, all during business hours, zero user impact. That profile argues for patience. The same failure with a 2 a.m. median page time flips the arithmetic entirely.

The cost line that settles most debates

Compute a monthly cost in plain units: recurrences per month × (MTTR in minutes × loaded engineer rate per minute) + user impact. At a $90/hour loaded rate, a minute costs $1.50. A failure paging four times a month at twenty-five minutes per recovery costs $150 in labor alone; add the user-facing minutes and the total usually lands between $200 and $400. Compare that against the fix estimate multiplied by 1.5 — the multiplier is the honest buffer for review, testing, and the regression you will cause. A sixteen-hour fix at $1,440 breaks even against a $300 monthly cost inside five months. Break-even beyond a year means workaround with a review date, not fix. The point is not precision; it is that the debate moves from adjectives to two numbers anyone can check.

Signals that point to a code fix

• It violates a guarantee you have written down. Backups restorable within your stated RPO, WAL archiving with no gap past your threshold, checkout completing in under two seconds. Guarantees outrank cost arithmetic; a workaround that masks a violated guarantee is an outage you have scheduled.
• The workaround requires a human to remember something. Runbooks reduce that cost but do not remove it; memory is not a control.
• The fix touches code you own and has reviewable scope. A diff under a few hundred lines that one engineer can write, review, and test inside a week qualifies.
• The failure delays or corrupts data. Anything discovered late — a two-day-stale backup found during a restore drill — counts double.
• You have patched the same seam twice and it regressed. Repeated patches at the same spot mean the design is wrong, not the patch.

Signals that point to an operational workaround

• The root cause is in a dependency you do not control. A vendor API returning 429s at month-end, intermittent mount behavior from a cloud provider, a flaky upstream resolver. You cannot patch code you do not own; you can absorb its behavior.
• The workaround is reversible configuration. PgBouncer pool sizing, a systemd timer restart, retry-with-backoff at the proxy, an archive_timeout value. Config with rollback is cheap to test and cheap to undo.
• The failure is rare relative to the fix. Once a quarter with a five-minute recovery does not justify a migration that consumes a month of one engineer’s capacity.
• The code fix requires a migration. Schema changes, queue rewrites, and storage swaps carry risk a small team absorbs poorly; the workaround buys time to plan them properly.

Three incidents, three verdicts

The archive gap that was not a code bug (PostgreSQL 16, pgBackRest 2.50)

A six-person team on GCP was flagged by a synthetic check for WAL archive gaps longer than fifteen minutes, two to three times a month. The instinct was a pgBackRest bug: the check measured the distance between the current WAL position and the last archived segment, and the gaps looked like data at risk. The evidence disagreed. Gaps clustered between 2 a.m. and 4 a.m., when write volume fell and segments were not filling fast enough to switch. The fix was operational in two parts: archive_timeout = 300 to force a switch every five minutes, and a corrected check threshold that told a slow segment apart from a stalled archiver. No application or backup code changed. Verdict: fix the monitoring and the configuration, not the software.

The vendor API nobody could patch (retry queue)

A three-person team integrating a tax calculation API saw 429 and 503 bursts during the vendor’s month-end peak. The root cause was demonstrably not theirs, the vendor would not commit to a fix date, and the failure was absorbable: requests were not time-critical inside a two-hour window. They wrote a forty-line worker that queued failed requests and retried them with exponential backoff, added a queue-depth alert, and documented the flow in the runbook. Verdict: a permanent operational workaround, reviewed quarterly, owned by name. This is what a good workaround looks like — an owner, an alert, and a review date.

The workaround that masked a broken guarantee (restic 0.16)

A two-node bare-metal team ran nightly restic backups on a systemd timer. Interrupted runs left stale locks, and the on-call developed a habit: run restic unlock by hand, acknowledge the page, move on. This went on for six weeks. Their monthly restore lottery — a drill that picks a random snapshot to restore and verify — then produced a snapshot two days older than expected, because the previous night’s run had failed on a lock and the alert had been acknowledged rather than investigated. The cost line had crossed weeks earlier; the drill made it visible. The fix was twenty lines: a pre-run lock check that unlocked only when no restic process was active, enforced with systemd unit ordering. Merged in an afternoon, reviewed by the second engineer, recurrence stopped. Verdict: code fix, overdue.

The pattern across the three: the verdict rarely matches the instinct. The suspected software bug was a threshold problem. The obvious code fix was impossible. The harmless manual habit was eroding the one guarantee the team had written down.

Workarounds need expiry dates

An undocumented workaround is a future incident with a delay timer. Treat every workaround as a decision with a review attached. The lightest format that works is a single page in the ops repository — a register with one row per workaround: the failure mode it absorbs, the named owner, the monitoring signal that detects when the workaround itself breaks, and a review date. A three-paragraph decision record next to each row keeps the reasoning findable: context (the recurrence count and cost line), decision (workaround, plus the conditions that would reverse it), and consequences (what you accept until the review date).

Two rules keep the register honest. First, no workaround ships without a runbook entry and a monitoring signal — a workaround you cannot detect failing is a blind spot, not a mitigation. Second, anything older than two quarters gets a re-decision rather than renewal by silence. The runbook entry deserves the same care as a recovery checklist written before it is needed; the discipline transfers directly.

Who makes the call, and when

Not during the incident. During an incident the on-call restores service with whatever is fastest and safest, and a workaround applied under pressure is legitimate. The fix-or-workaround decision belongs to the postmortem’s action-item review, within a week, with recurrence data on the table. Google’s SRE book argues that blameless postmortem culture produces better action items than accountability-driven reviews, and Etsy’s 2012 piece on blameless postmortems remains the clearest short statement of why: engineers who expect blame report less, and unreported failures compound.

In a team of two to fifteen, the decision meeting is thirty minutes with two people: the on-call who handled the last occurrence and the engineer who owns the service area. The output is the decision record, not a consensus memo. When those two disagree after looking at the same cost line, the tiebreaker is the guarantee question — does the failure violate something you have promised in writing, to yourself or a customer? Guarantees win.

The procedure, end to end

The whole method compresses into seven steps small enough to run during one ops review:

1. Tag the failure mode and count recurrences from the last ninety days of paging history.
2. Record MTBF and MTTR from incident timestamps; note time-of-day and user impact for each occurrence.
3. Compute the monthly cost line: recurrences × (MTTR in minutes × loaded rate) + user impact.
4. Estimate the code fix in engineer-hours, multiply by 1.5, and flag anything unbounded — migrations, schema changes, vendor coordination.
5. Answer the two qualifying questions: do you own the code that produced the failure, and does the failure violate a stated guarantee?
6. On the fix path, schedule the change inside two sprints and link the decision record to the postmortem.
7. On the workaround path, write the runbook entry, add the monitoring signal, set the review date, and add the row to the register.

For a shorthand version, the signals compress into one table:

Signal	Leans toward	Why
Three or more pages in ninety days	Code fix	Cost compounds, and alert fatigue is real
Root cause in a vendor or dependency	Workaround	You cannot patch code you do not own
Workaround needs a human to remember a step	Code fix	Memory is not a control
Failure violates a written RPO or RTO	Code fix	Guarantees outrank estimates
Fix requires a multi-week migration	Workaround with review date	Capacity is finite; buy time deliberately
Workaround is reversible config	Workaround	Cheap to test, cheap to undo

Frequently asked questions

How many recurrences justify a code fix?

Three in ninety days is the working threshold for a small team, because three is enough to compute a meaningful MTBF and enough to prove the first fix did not hold. Impact modifies the count: one recurrence that breached your RPO during a restore drill outranks five benign warnings, and a failure mode that pages at night counts double. Treat the threshold as the entry point to the cost line, not a substitute for it.

What separates a workaround from a band-aid?

Evidence. A workaround is documented in the runbook, detected by a monitoring signal when it breaks, owned by a named person, and reviewed on a date. A band-aid is none of those things. The test: if a new on-call engineer hit the failure tonight, could they find the workaround in the runbook and know whether it is still sanctioned? If not, it is a band-aid, and it is accruing debt at interest.

How do you keep workarounds from becoming permanent?

Expiry dates and a monthly read-through. Every workaround gets a review date at creation — ninety days is a reasonable default — and the register is read aloud in the ops review. Anything past its date gets an explicit re-decision: fix it, extend it with a written reason, or accept it permanently and promote it into the runbook as normal operating procedure. Silence never renews a workaround.

Should the on-call engineer make this call during an incident?

No. During an incident the on-call restores service with whatever is fastest and safest, including a workaround, and that judgment is not second-guessed afterward. The fix-or-workaround decision happens in the postmortem review within a week, with the recurrence data in hand. Deciding mid-incident optimizes for adrenaline; deciding in the review optimizes for the pattern.

When is a workaround better than a code fix even though you could fix the code?

Three situations recur in practice: the root cause sits in a dependency whose behavior you can absorb but not change; the code fix requires a migration whose risk exceeds the failure’s cost; or the failure is rare — roughly once a quarter with a five-minute recovery — so the fix would be the most expensive line item of the quarter for that service. In each case the workaround is a deliberate, documented, reviewed decision rather than a deferral.

Where to start this week

Export the last ninety days of pages, tag the top three failure modes by count, and run the cost line on each. Take the one with the worst ratio of monthly cost to fix estimate and write its decision record — three paragraphs, thirty minutes. If the verdict is a workaround, the runbook entry and its monitoring signal come first; the template we use is the one in Write the Recovery Checklist Before You Need It. If the verdict is a fix, schedule it before the next occurrence argues with you.

Recurring failures are the most legible data a small team has about where its systems actually hurt. Deciding fix-or-workaround on a cadence, with two numbers and two qualifying questions, turns that data into the only kind of reliability progress a team without an SRE department can sustain: small, boring, and repeatable. If you run the cost line on a real failure and disagree with your own verdict, send us the anonymized numbers — working reader verdicts is the column we want to run next.