The short answer

Price the failure, price the fix, and compare them over a 90-day horizon. If the workaround costs more than the fix within two quarters, fix the code. If the fix is riskier than the failure, or the failure is upstream and already corrected in a newer release, take the workaround — then give it an owner, a runbook step, and a review date. One exception overrides the math: a recurring failure on a recovery path — backups, WAL archiving, restore tooling — gets a code fix regardless of cost, because those systems fail silently until the day you need them.

These field notes come from Gray Haven Lab’s recovery drills with teams of two to fifteen engineers, and most of the numbers below surfaced in those drills. The rest is the working method: the three numbers to collect, the signals that tip the decision either way, a worked example from a six-engineer team, and a register format that keeps accepted workarounds from becoming permanent debt.

Definitions first, because the argument usually starts here

A code fix is a change to the service, its configuration, or its dependencies that removes the failure mode: moving restic from 0.15.1 to 0.16, adding retry with backoff to a PostgreSQL archive_command wrapper, or correcting a pool setting that lets standby replication lag grow. An operational workaround is a procedure that absorbs the failure without changing what runs in production: a runbook step, a synthetic check, a scheduled cleanup job, a PagerDuty escalation tweak, a manual rerun.

Both are legitimate engineering outputs. Most team arguments here are not about whether to fix things; they are about what to fix first when the roadmap is full and the on-call rotation is two people deep. That argument is easier with numbers.

The decision math: three numbers and a rule

Collect three numbers before debating anything:

• Cost per occurrence. Toil minutes, plus blast radius. A failed nightly backup that takes 12 minutes to rerun costs 12 minutes. The same failure paging someone at 2 a.m. costs closer to 45 minutes once you count the wake-up, the context switch, and the sleep debt.
• Recurrence interval. Days between occurrences, from your incident log. Trend it if you can: 14 days, then 9, then 6, is a curve, not a pattern.
• Fix effort and fix risk. Engineer-days to ship, and the worst plausible outcome of shipping it.

The rule of thumb: quarterly toil = (90 ÷ recurrence interval) × cost per occurrence. When quarterly toil exceeds fix effort within about two quarters, the fix pays for itself. When fix risk exceeds the failure’s blast radius — a schema migration on a 900 GB production table to silence a recurring warning, say — the workaround wins, at least until the migration can be staged with a dry run and a tested rollback.

Run a concrete case. A restic 0.15.1 job fails every 10 days and takes 12 minutes to rerun: nine occurrences a quarter, 108 minutes of toil. The fix — moving to 0.16, where the lock-handling defect is corrected — costs about half a day. Payback lands at roughly two quarters on toil alone, and sooner once you count the alert noise and the 2 a.m. variants. That is a fix. Now the same failure every 70 days: 1.3 occurrences a quarter, 16 minutes of toil. Half a day of engineering against 16 minutes a quarter is not a fix; it is a runbook step and a scheduled rerun.

Four signals the failure deserves the code fix

1. The failure sits on a recovery path

Backups, WAL archiving, and restore tooling fail without noise: green dashboards, no restore. A recurring failure there is compound risk, because the day you need the system is the day a stale workaround is least likely to work. If pgBackRest 2.47 archive-get calls time out and the standby drifts, you fix the archive path — retries, credentials, network — rather than adding a “restart the standby” step and calling it done. The PostgreSQL documentation on continuous archiving is blunt about the stakes: recovery stops at the first missing segment in the WAL chain, so an archive gap is a restore that ends early, whether or not anyone noticed the gap.

2. The recurrence interval is closing

Workaround cost is linear; failure growth often is not. Pull the interval from your last three postmortems. If the gap is closing, the failure you priced last quarter is now a different, more expensive failure. Fix it while it is still a small fix.

3. The workaround needs judgment at 3 a.m.

There is a difference between a runbook step that says “rerun job X and confirm exit code 0” and one that says “check replication lag; if it exceeds 15 minutes, decide whether to fail over.” The first is a procedure. The second is an incident waiting for a tired human. A workaround that moves the failure from business hours to the on-call rotation has not reduced the risk; it has relocated it.

4. The failure is spending your error budget

If recurrences breach the SLO often enough to burn the budget, the decision is already made — error budgets exist so that reliability work trades against roadmap work on explicit terms. A workaround that keeps the service inside its SLO can be an honest answer, but only if it is counted, owned, and reviewed like the debt it is.

Four signals the workaround is the right call

1. The fix is riskier than the failure

Some fixes carry more blast radius than the failure they remove. A schema migration on a 900 GB production table to stop a duplicate-row warning is a riskier change than a weekly five-minute cleanup job. When the fix’s worst case beats the failure’s worst case, take the workaround and stage the fix properly: dry run, canary, rehearsal against a restore.

2. The failure is upstream and versioned away

When the defect lives in a dependency and the correction already ships in a newer release, the workaround is a bridge, not debt. restic’s stale-lock behavior on interrupted S3 backups, corrected in the 0.16 series, is the standard shape: pin the rerun step to the runbook, schedule the upgrade, and write the version number that retires the workaround into its register entry.

3. The failure is rare, contained, and cheap

A staging-cluster certificate hiccup that costs 10 minutes a quarter does not need two days of Terraform work. Rare, cheap, contained failures can live as runbook steps indefinitely — provided the steps still get exercised, which is exactly what a restore lottery is for.

4. You are still in the incident

During the incident, the workaround is always the right first move. The decision this article is about happens in the two weeks after, in the post-incident review. If nobody makes it explicitly there, the workaround becomes the system by default.

A worked example: the WAL gap that came back three times

Incident 2024-11-06, details anonymized, numbers real. A six-engineer team running PostgreSQL 15 on EC2, with pgBackRest 2.47 archiving WAL to S3 and one streaming standby in a second region. During a VPC endpoint migration, archive_command calls began timing out; the standby drifted 40 minutes behind; a human caught it on a replication-lag graph, not an alert.

First occurrence: 25 minutes of toil, a manual archive-restart step, no ticket. Second occurrence, 19 days later: 30 minutes, the same step, a ticket filed with no owner. Third occurrence, 11 days after that: 45 minutes at 06:10, because this time the standby had drifted far enough to matter and the rerun did not take on the first try.

At the third occurrence the team ran the math. Extrapolating the closing interval, quarterly toil was heading past 200 minutes and rising. Judgment load was growing — the rerun step had already gained a conditional branch. Fix effort was roughly two days. They shipped retry with backoff in the archive wrapper, moved the archive host to instance-profile credentials so endpoint changes stopped breaking the chain, and added a synthetic check on replication lag with a 15-minute threshold so discovery stopped depending on someone opening a graph.

The fix survived. The synthetic check stayed, because a check that catches drift early is good operations whether or not the failure exists. The runbook step was retired one full recurrence interval later, after the fix had also survived a restore drill.

The pattern worth copying is not the fix; it is the decision point. First occurrence: runbook step. Third occurrence inside 60 days: a recorded fix-or-workaround decision with numbers attached. Teams that skip the decision point end up with runbooks full of orphaned steps nobody remembers the reason for — the exact failure that keeping runbooks alive through offboarding is meant to prevent.

Keeping workarounds from becoming permanent debt

A workaround is a loan against future engineering time, and loans need terms. Keep a register row for each accepted workaround with five fields:

• Trigger: the alert or symptom that invokes it.
• Runbook step: the link, not a paraphrase.
• Owner: one named engineer, not “the team.”
• Review date: quarterly, next to the error budget review.
• Retirement condition: the version, fix, or event that makes it obsolete.

Two habits keep the register honest. First, fold workarounds into the restore lottery: when a drawn drill touches a system carrying an accepted workaround, run that workaround’s runbook step as part of the drill and record whether it still works. Second, record the fix-or-workaround decision in the postmortem itself, so the next responder knows why the step exists and what would retire it. A step whose origin nobody can explain is a step that gets skipped at 3 a.m.

The middle path: shrink the failure while you schedule the fix

Between “fix it now” and “live with it” sits a third move: reduce the failure’s cost while the fix waits for roadmap space. Cut cost per occurrence with a rerun script or a synthetic check that catches drift before a human does. Cut recurrence with a circuit breaker or jitter on retries. These are still workarounds — they change the math, not the failure mode — but they buy the roadmap time without asking the on-call rotation to absorb a growing cost. Used deliberately, with a register entry and a review date, the middle path is engineering. Used by default, it is how a two-person team ends up running forty undocumented procedures from memory.

Frequently asked questions

How many recurrences justify a code fix?

Three in 60 days is a useful trigger because it separates pattern from noise, but recurrences are evidence, not the decision. The decision is quarterly toil versus fix effort. Three cheap, contained recurrences can live honestly as a runbook step; one judgment-heavy recurrence on a recovery path can justify the fix immediately.

Is a runbook step a fix?

No. A runbook step makes the failure survivable, not absent. It is the right first move, and it is also the evidence you will use when the fix-or-workaround decision comes due — a second reason to write it down properly the first time.

What if there is no roadmap space for the fix?

Then there is no roadmap space for the failure either; the toil is already being paid, in minutes instead of days. Put the workaround in the register, put the fix on the backlog with an estimate attached, and let the quarterly review date force the tradeoff back into view.

How do you tell when a workaround is hiding a bigger failure?

Watch its shape. If the runbook step keeps gaining conditional branches, the alert threshold keeps widening, or the recurrence interval keeps closing, the underlying failure is growing. A stable workaround looks boring from every angle.

When should a workaround be retired?

When the fix ships and survives one full recurrence interval plus one restore drill. Retiring the step on ship day is how teams discover that the fix worked in staging while the deleted runbook step was quietly load-bearing in production.

Decide once, write it down

The goal is not zero workarounds. A team of two to fifteen that ships will always carry a few. The goal is workarounds with owners, runbook steps that still work when tested, and fixes with dates — a system in which the decision gets made once, deliberately, and written down where the next on-call can find it.