…

…”
}

Let me write it all out now, carefully.

Also, minor: “2 a.m.” — keep. “Backblaze B2” keep. “pgBackRest 2.50” keep. “restic 0.16.4” keep. All factual claims preserved.

One small humanization: “We have been burned both ways.” — I changed to “we have been burned both ways” — keep contraction “we’ve been burned both ways”? Contraction reads more natural: “and we’ve been burned both ways.” Good.

Also in the intro, “hands you the same fork” — good.

I think the piece is ready. Write final JSON, single html string.{
“title”: “Code Fix or Workaround: How to Decide When a Recurring Failure Deserves the Real Repair”,
“html”: “

Every lean team collects failures that refuse to stay dead. A nightly restic job dies on a stale lock. A report endpoint exhausts its connection pool every second Friday. A managed queue throttles without so much as a warning. Each recurrence hands you the same fork: repair the cause in code and configuration, or absorb it with an operational workaround — a scripted step, a runbook line, an alert threshold that keeps the symptom at arm’s length. We call this the fix-or-workaround decision, and on a team of two to fifteen engineers with no dedicated SRE function, nobody owns it by default. It gets made by whoever holds the pager at 2 a.m. Tired, and reaching for the fastest exit. That default is expensive in both directions, and we’ve been burned both ways. The wrong workaround becomes invisible toil that outlives its author. The wrong fix burns scarce engineering time on a failure mode that never earned it. This field note is the framework we use instead — five evidence signals, two anonymized incidents, and a one-page decision record that keeps the choice from quietly expiring.

What counts as a recurring failure — and what does not

A recurring failure is one failure mode observed three or more times — not three failures that merely look alike. The distinction matters, because the fix-or-workaround decision only pays off when you’re confident the occurrences share a mechanism. Our test is mechanical. Every incident note gets a failure-mode tag — backup-lock-stall, pool-exhaustion-report — and the tag only goes on if the responding engineer can name the mechanism, not just the symptom. Three separate 5xx spikes with three different causes are three first-time incidents. Three spikes where the same report endpoint holds PostgreSQL connections open past its statement timeout are one recurring failure with a growing bill.

An operational workaround lives outside your codebase: a runbook step, a wrapper script, a scheduled job, a raised threshold, a bigger instance. It compensates for the failure mode without removing it. A code fix removes the failure mode at its source — application code, schema, infrastructure configuration, or a dependency version bump. The tradeoff never changes, so we’ll state it plainly. Workarounds are cheap to install and cheap to reverse, but they charge operator attention on every recurrence. Fixes are expensive to install, carry rollout risk, and are usually permanent. Neither choice is virtuous by default. A team that only ever works around accumulates toil it can’t see. A team that only ever fixes spends its scarcest budget on failure modes that may not deserve it.

The five signals we score before deciding

When a tagged failure mode hits its third occurrence, we spend about thirty minutes scoring five signals. The score doesn’t decide for you. What it does is put the tradeoffs on the table before someone burns a weekend writing a fix nobody asked for — or before a workaround quietly becomes one person’s muscle memory and nobody else’s.

1. Recurrence cost, in operator minutes

Multiply frequency by handling time over a rolling month. A failure that recurs twice monthly and takes twenty minutes of on-call effort to clear costs about forty operator minutes a month. If a credible fix runs three engineer-days — roughly 1,440 minutes — the fix pays for itself in handling time alone after about three years. Weak case, unless another signal intervenes. Our working thresholds, tuned on small teams: below roughly thirty operator minutes a month, a workaround is almost always right; above two hours a month, the fix gets scheduled regardless of anything else; between those two, blast radius decides. Treat the arithmetic as a floor on the fix’s value, not a ceiling. It deliberately undercounts interrupted sleep, context switching, and the drift that sets in when the same person is always the one who knows the incantation.

2. Blast radius: does it touch the data path?

Failures that only degrade convenience tolerate workarounds well. Failures that can corrupt, truncate, or lose data do not. Our hard rule: any recurrence with a plausible data-integrity path — a WAL archive gap, a backup that fails verification, a write path that can partially commit — earns either an immediate fix or a workaround that is itself alarmed. A compensating control that can fail silently isn’t a control. PostgreSQL point-in-time recovery depends on an unbroken WAL chain, whether that chain is managed by pgBackRest 2.50 or by native continuous archiving, as the PostgreSQL documentation states plainly. An archiving stall that a cron job clears by restarting the archiver is acceptable only if a second, independent check confirms the chain actually resumed. A silently gapped archive is exactly the failure class a restore lottery drill exists to catch — before a real restore does.

3. Where the ownership boundary sits

If the defect lives in a dependency you don’t maintain, the code fix isn’t yours to write. The decision narrows to four levers: pin a version, change a configuration, open an upstream issue with your evidence attached, or wrap the boundary with a script. The stale-lock behavior in restic belongs to restic. A team running restic 0.16.4 manages that behavior; it does not repair it. When the failure mode crosses into code you own, the real fix is on the table, and the cost estimate in signal five begins.

4. Whether the workaround can be observed

A workaround nobody can see becomes invisible toil. It runs in one person’s memory, dies with an offboarding, and resurfaces months later as a mystery outage. Every workaround we register gets at least one of three signals — a synthetic check, a metric, or a low-urgency PagerDuty alert that fires when the workaround is invoked. If a candidate workaround can’t be made to emit any trace, that isn’t grounds for instant rejection, but it is evidence for the fix. Google’s SRE workbook recommends keeping toil below half of an engineer’s time before it crowds out project work, and you can’t manage that budget against workarounds you can’t count.

5. What the fix actually costs

Estimate the fix as writing, review, deployment, a rollback plan, and verification — not as the diff. On a four-person team, a two-line change to a billing-critical path is not a two-line change. It’s a release with money on the line and no second reviewer awake. Ask what the fix can break, too. A pool ceiling that stops exhaustion can convert hard 500 errors into slow, soft queuing that nobody notices for a week. Fixes have blast radii, and an estimate that ignores them is marketing, not arithmetic.

Two incidents, two different answers

Same fork, opposite verdicts. Here is what tipped each call.

The stale restic lock: the workaround won

A six-person B2B scheduling team we advised runs nightly restic 0.16.4 backups to Backblaze B2 over a WAN link that drops a few times a month. An interrupted run occasionally left a lock behind, and the next night’s job failed with repository is already locked — an error that looks like a security incident at 2 a.m. and almost never is one. Recurrence: roughly every six weeks, about twenty-five minutes of on-call handling, no exposure to the data path. And the fix did not exist to be written. Lock semantics belong upstream, so signal three removed the code fix from the menu entirely.

They wrote a forty-line wrapper instead. Verify no live restic process, run restic unlock, retry the backup, emit a metric, and open a low-urgency PagerDuty alert if two consecutive nights fail. Recurrence cost dropped from twenty-five operator minutes per event to zero pages and about one ticket a quarter. The decision record reads: workaround is the permanent answer; revisit at the next restic minor release; reopen if the alert fires more than once a quarter, because rising frequency is the tell that this arrangement is rotting.

The Friday pool exhaustion: the fix won

A four-person fintech team saw 5xx bursts roughly every nine days, cleared by an app-tier restart in about fifteen minutes. The arithmetic said workaround: about sixty operator minutes a month against a fix estimated at two engineer-days, and the mean time to restore looked respectable on the monthly report. That’s part of why the restart step survived a full quarter. Two signals overrode the arithmetic. The blast radius touched the data path — a report query held connections open long enough to starve transactional traffic — and the workaround was masking a trend: the interval had shortened from fourteen days to nine inside a single quarter. A recurring failure whose interval is shrinking is a fix candidate regardless of what the cost model says, because the model prices today’s frequency, not next quarter’s.

The fix was unglamorous. A statement timeout on the report path, a pool ceiling, and a Grafana panel on pool saturation so the trend stayed visible after the symptom disappeared. Two engineer-days, one rollback plan, and a canary deploy on a Friday morning rather than a Friday evening. The team also retired the restart runbook step that same week — a workaround and its fix shouldn’t coexist past the deploy that makes the workaround dead weight.

Write the decision down, or you never made it

A fix-or-workaround decision that lives in a Slack thread is a decision that expires silently. We keep a one-page decision record per recurring failure, stored beside the runbooks, with six fields: date, failure-mode tag, evidence (occurrence count, operator minutes, trend), the decision itself, the expiry trigger that will reopen the question, and an owner. It pairs naturally with the habit of writing the recovery checklist before you need it — every registered workaround is a line item on that checklist, and a checklist step with no owner and no alert is the same silent toil in a different binder.

When the decision lands on the workaround side, run a fifteen-minute pre-mortem before closing the record. The prompt is one sentence: it is eight months from now, and this workaround just made an incident worse — what happened? The answers tend to be specific. The wrapper script never made it into the new-hire runbook. The low-urgency alert got muted during a release and never unmuted. The unlock step ran against the wrong repository after a storage migration. Each answer becomes a guardrail line in the record.

Expiry triggers we use — any one of these reopens the decision:

• Recurrence crosses the two-hour monthly operator threshold.
• The workaround fires during a customer-visible incident. Even once.
• A new hire can’t execute the runbook step in a single pass during onboarding.
• Upstream ships a fix, or the pinned dependency approaches end of life.
• The registry doubles in size within a quarter — a system-level sign the team is deferring rather than deciding.

Audit the registry quarterly. The count of active workarounds is a health metric, not a trophy. A small number with alerts and owners is normal operations. A growing number with neither is debt compounding at on-call interest rates.

When the workaround is the correct final answer

Some failure modes should never get a code fix, and saying so out loud keeps the registry honest. Four cases recur in our notes. First, upstream-owned defects, where your whole lever is a version pin and an issue number. Second, systems scheduled for replacement: fixing a report path you plan to delete next quarter is investment in the wrong asset, and a documented workaround with an expiry date is the disciplined answer. Third, low-frequency failures with a high fix cost where the blast radius is fully covered by an alarmed control — a synthetic check that catches the degradation in under a minute can make a quarterly manual step acceptable. Fourth, vendor-managed limits and quotas, where the only lever you actually own is the alert that tells you before the customer does.

The cautious framing matters more than the list, though. Workarounds compound. Each one costs a little attention forever, and attention is the one budget a team of five can’t print. A workaround that was right in March can be wrong by October because headcount, traffic, or the dependency changed underneath it. That’s why the record carries an expiry trigger and the registry carries an audit date. The decision is renewable, not permanent, and renewing it costs thirty minutes.

The habit, compressed

Tag the failure mode. At the third occurrence, score the five signals: operator minutes, blast radius, ownership, observability, and true fix cost. Decide. Record the decision with an owner and an expiry trigger, and pre-mortem any workaround you keep. The whole ceremony fits inside a lunch break, and it replaces the two expensive defaults — the 2 a.m. reflex and the weekend fix nobody scoped.

Frequently asked questions

How many recurrences justify a code fix?

Three verified occurrences is the entry bar for making the decision, not a mandate to fix. A recurrence with a plausible data-integrity path can justify a fix on the first occurrence; a cosmetic twenty-minute failure can recur monthly for years and still lose to a well-alarmed workaround. The count gets you into the room. Operator minutes and blast radius hand down the verdict.

How do you keep workarounds from becoming invisible toil?

Register each one, make its invocation emit a signal — a metric, a synthetic check, or a low-urgency PagerDuty alert — and audit the registry quarterly with expiry triggers attached. A workaround that leaves no trace when it runs is the definition of invisible toil, and the inability to instrument it is itself evidence for writing the fix.

What if the fix belongs to a dependency you do not maintain?

Then the decision is about the boundary, not the interior. Pin the version, change the configuration, open an upstream issue with your occurrence data attached, and wrap the boundary with a script that absorbs the symptom. In the restic example, the wrapper script is the entire fix available to that team, and recording it as a deliberate boundary decision beats pretending a deeper repair is on the roadmap.

Does every recurring failure need a full postmortem?

No. The first occurrence gets an incident note; the second gets the failure-mode tag and a five-minute check that the first note named the right mechanism; the third gets a full blameless postmortem with action items tracked to closure. Google’s SRE book makes the case that postmortems only pay off when action items are followed through, and a recurring failure is exactly the case they were built for. The action item list from that third postmortem is your fix-or-workaround queue, pre-sorted by evidence.

Got a recurring failure sitting on the fence — one where the arithmetic says workaround and your gut says fix? Send us the details, anonymized. Scoring reader-submitted calls is becoming a regular column here, because the fastest way to calibrate a framework is to run it against failures we didn’t choose ourselves.