Why the Fix-or-Workaround Decision Matters for Lean Teams

A recurring failure is any incident that returns with the same signature: the same alert, the same service, the same approximate time window, the same customer-visible symptom. For a team of two to fifteen engineers who own their own on-call rotation, every recurrence forces a decision that has less to do with engineering taste and more to do with where the next hour of reliability budget goes. The choice is rarely binary. It is a portfolio decision: fix the code, absorb the failure with an operational workaround, or do both in sequence.

This article is a decision framework, not a mandate. It assumes you run production on AWS, GCP, or bare metal without a dedicated SRE department. It assumes your monitoring is PagerDuty or a comparable escalation tool, your backups are restic or pgBackRest, and your runbooks live somewhere a tired engineer can find them at 03:00. The goal is to make the fix-or-workaround call explicit, repeatable, and reviewable, so the same failure does not quietly consume a quarter of your on-call capacity.

Define the Failure Before You Define the Fix

Most bad decisions start with a vague incident description. “The database was slow again” is not a failure definition. A useful definition includes the trigger, the blast radius, the detection path, and the recovery path. Write it in one paragraph and keep it in the incident record.

The four fields that make a recurrence comparable

• Trigger: What changed or accumulated before the failure? A cron job at 02:15, a batch import, a certificate rotation, a traffic growth rate above 20% week over week.
• Blast radius: Which services, regions, or customer segments degraded? One availability zone, one read replica, one tenant.
• Detection: Which synthetic check, growth-rate alert, or PagerDuty escalation rule caught it? If a human reported it first, that is a detection gap worth naming.
• Recovery: What actually restored service? A restart, a failover, a restore from pgBackRest, a manual WAL replay, a config rollback.

Once two or more incidents share these four fields, you have a recurrence. Until then, you have noise. The distinction matters because workarounds applied to noise create permanent complexity for no reliability gain.

The Decision Framework: Five Questions in Order

Ask these questions in sequence. Stop at the first one that gives a clear answer. The order is deliberate: it front-loads the cheapest and most reversible options.

1. Is the failure already contained by an existing control?

If a runbook step, a circuit breaker, or a scheduled maintenance window already prevents customer impact, the recurrence may be acceptable. A nightly vacuum that briefly raises replica lag, caught by a synthetic check and absorbed by read routing, is not a code bug. It is a known cost. Document it, set a review date, and move on. The trap is treating every alert as a defect. Some alerts are the system telling you it is working as designed.

2. Does the workaround have a bounded lifetime?

Operational workarounds are legitimate when they are temporary and dated. A manual step in the runbook that says “restart the worker pool after the 02:15 batch” is acceptable for one quarter if the batch job is being replaced. It is not acceptable indefinitely, because it transfers reliability risk from the code to the on-call engineer’s memory. If the workaround has no expiry date, it is not a workaround. It is technical debt with a pager attached.

3. What is the cost of the code fix relative to the recurrence rate?

Estimate both sides in the same unit: engineer-hours per quarter. A failure that costs 45 minutes of on-call time twice a month is roughly 18 hours per quarter, plus context-switching and postmortem overhead. A code fix that takes 30 hours to design, implement, and verify pays back in under two quarters. A fix that takes 120 hours and touches a shared library used by six services may not pay back within the year, especially if the failure is cosmetic or self-healing.

Use your incident records, not intuition. If you do not have incident records, start with a simple table in your runbook repository: date, duration, detection source, recovery action. Three months of that table will change how you prioritize.

4. Does the fix reduce a class of failures or just one instance?

A code fix that eliminates one alert but leaves the underlying pattern intact is often worse than a workaround, because it creates false confidence. A fix that adds idempotency to a job runner, or that makes a retry policy respect backoff, may close a whole class of recurrences. Prefer fixes that generalize. When you cannot generalize, prefer workarounds that are visible and dated.

5. Can the workaround be tested like code?

If the workaround is a runbook step, it should be rehearsed. If it is a script, it should live in version control and run in a staging environment. If it is a manual database intervention, it should be paired with a restore drill so the team knows the recovery path is real. A workaround that has never been rehearsed is not a control. It is a hope. The Recovery Checklist Before You Need It is a useful template for turning ad hoc recovery steps into rehearsed procedures.

When a Workaround Is the Right Answer

Workarounds are not failures of engineering discipline. They are appropriate when the failure is rare, the impact is contained, the fix is expensive, and the workaround is observable. Three patterns justify a workaround:

• Third-party dependency behavior: An upstream API returns intermittent 503s during its maintenance window. You cannot fix their code. You can add a retry with jitter and a synthetic check that distinguishes their maintenance from your outage.
• Infrastructure-level noise: A cloud provider’s instance retirement notices cause a brief spike in your PagerDuty queue. The workaround is a suppression rule scoped to that event type, reviewed quarterly.
• Low-frequency, high-cost fixes: A failure occurs once every 18 months and the fix requires a schema migration across a multi-terabyte PostgreSQL cluster. The workaround is a documented manual failover with a pgBackRest restore path, rehearsed twice a year.

In each case, the workaround has a named owner, a review date, and a detection signal. Without those three, it drifts into folklore.

When a Code Fix Is the Right Answer

Code fixes earn their cost when the failure is frequent, the workaround is fragile, or the fix closes a class. Signals that point toward a fix:

• The same alert fires more than once per month and each occurrence consumes more than 30 minutes of on-call time.
• The workaround requires a specific engineer’s knowledge, and that engineer is approaching a vacation or role change.
• The failure has a customer-visible symptom, even if brief, and the workaround depends on a human noticing it.
• The fix is localized: one service, one module, one configuration path, with a clear test that reproduces the failure.

When you choose a fix, write the decision record. Name the failure signature, the recurrence rate, the estimated fix cost, and the expected reduction. This is the same discipline as a blameless postmortem, applied before the work rather than after. A pre-mortem that asks “what would make this fix fail to reduce recurrences?” is cheaper than discovering it six weeks later.

Sequencing: Workaround First, Fix Second

For lean teams, the most common correct answer is both, in order. Ship the workaround this week to stop the bleeding. Schedule the fix for the next planning cycle. The workaround buys time; the fix buys capacity. The failure mode to avoid is shipping the workaround and never scheduling the fix, because the workaround made the pain invisible.

Make the sequence explicit in your runbook: workaround owner, fix owner, review date. If the fix slips, the review date forces a conversation rather than a silent deferral.

Access Hygiene and Break-Glass in the Decision

Some recurrences are not code or operations problems. They are access problems. A failure that requires a break-glass credential to resolve is a signal that your normal access path is insufficient. If the same break-glass account is used twice in a quarter, the recurrence is in your access model, not your application. Onboarding and offboarding hygiene, scoped roles, and time-bound break-glass credentials are part of the fix-or-workaround decision because they determine who can act at 03:00 and how much damage a mistake can cause.

Document break-glass use in the incident record. If the pattern repeats, treat it as a design input, not an operational quirk.

Monitoring and Failure Rehearsal as Decision Inputs

You cannot decide whether a recurrence deserves a fix if you cannot measure it. Synthetic checks, growth-rate alerts, and PagerDuty escalation rules are the instruments that turn a vague pattern into a comparable record. A growth-rate alert that fires when disk usage increases more than 15% in 24 hours is more useful than a static threshold, because it catches the trend before the outage. A synthetic check that exercises the restore path is more useful than a backup success notification, because it tests recovery rather than storage.

Rehearse the failure. If the workaround is a manual failover, run it in staging. If the fix is a retry policy, inject the failure and confirm the retry. Rehearsal converts assumptions into evidence, and evidence is what makes the fix-or-workaround call defensible.

FAQ

How many recurrences justify a code fix?

There is no universal number. A practical threshold for a lean team is two occurrences in a quarter with more than 30 minutes of on-call time each, or any occurrence with customer-visible impact and a fragile workaround. The threshold should be written down and reviewed, not rediscovered each time.

Can a workaround ever be permanent?

Yes, if it is bounded, observable, and cheaper than the fix over a multi-year horizon. A permanent workaround should have an owner, a review date, and a detection signal. If it lacks those, it is not permanent. It is unmanaged.

What if the fix is in a third-party dependency?

Then the decision is about mitigation, not repair. Add retries with backoff, synthetic checks that distinguish their maintenance from your outage, and a documented fallback. Track their incident history and review your mitigation quarterly. You cannot fix their code, but you can reduce your exposure to it.

How do we avoid workaround drift?

Put every workaround in the runbook with a date and an owner. Review the runbook quarterly. If a workaround has no review date, it is drift. The same discipline that keeps your backup and recovery drills current keeps your workarounds honest.

What to Do Next

Pick one recurring failure from the last quarter. Write its four fields: trigger, blast radius, detection, recovery. Ask the five questions in order. Record the decision, the owner, and the review date. Then rehearse the chosen path, whether it is a code fix or a workaround, so the next occurrence is a known procedure rather than a fresh incident. The decision framework is small. The discipline of applying it consistently is what keeps a lean team’s on-call rotation sustainable.