An on-call rotation is a scheduling system that assigns one or more engineers to respond to production incidents outside normal working hours. Adjacent concepts include escalation policy, alert fatigue, pager load, follow-the-sun coverage, and incident severity levels. For a lean technical team of two to fifteen engineers running cloud-native infrastructure on AWS, GCP, or bare metal, the rotation is often the only line of defense between a failed database backup and a customer-visible outage. The difference between a drained rotation and a sustainable one is not the number of alerts. It is whether the rotation produces learning, preserves sleep, and remains staffed without heroics.

Gray Haven Lab has worked with teams that carried pagers for years without a single documented post-incident review. Those teams did not fail because they lacked monitoring. They failed because every page became a one-off surprise. A sustainable rotation treats on-call as a feedback loop: alert, respond, document, reduce, repeat. A drained rotation treats on-call as a cost center where the same noisy alert wakes someone at 3 a.m. for the sixth time.

What a Drained Rotation Looks Like

A drained rotation has a recognizable shape. The same two engineers absorb most pages. The alert threshold for CPU usage sits at 80 percent on a host that routinely runs at 78 percent during batch jobs. The runbook says “restart the service” but does not say which service, which namespace, or how to verify recovery. After a week of pages, nobody writes down what happened because the team is already behind on planned work.

Common signals include:

• Pager load above 25 percent per on-call engineer. Pager load is the share of time an on-call engineer spends responding to pages. Google’s SRE book describes keeping pager load below 25 percent so engineers retain time for engineering work. Above that, the rotation becomes a second full-time job.
• Alert count rising while mean time to acknowledge stays flat. If the team acknowledges alerts quickly but never reduces them, the rotation is absorbing noise instead of eliminating it.
• No documented follow-up for pages that did not require action. A page that fires and resolves itself is not harmless. It trains the responder to ignore the next page.
• Rotation gaps during weekends or holidays. A lean team that loses one engineer to vacation should not lose on-call coverage. If it does, the rotation depends on a single point of failure.

Drained rotations also show up in access hygiene. When an engineer leaves and their credentials still work in the production AWS account, the remaining team inherits a silent risk. The on-call burden grows because the team cannot trust its own access boundaries. A sustainable rotation includes a quarterly access review tied to the on-call schedule, not a yearly audit that nobody reads.

What a Sustainable Rotation Looks Like

A sustainable rotation has three properties: predictable load, actionable alerts, and a written path from page to prevention. Predictable load means the team can forecast how many pages a typical week produces. Actionable alerts mean every page requires a human decision. A written path means the responder can open a runbook, follow it, and leave notes that feed the next incident review.

For a team on AWS, a sustainable rotation often starts with Amazon CloudWatch alarms scoped to customer-facing symptoms, not infrastructure internals. An alarm on HTTP 5xx rate from an Application Load Balancer is more useful than an alarm on EC2 CPUUtilization for a service that runs fine at high CPU. For GCP, the same logic applies to Cloud Monitoring uptime checks and request-based SLOs. For bare metal, Prometheus alert rules should page on error budgets, not on raw disk usage that fluctuates with log rotation.

Sustainable rotations also separate alerting from logging. A page should mean “a user is affected or about to be affected.” A log entry should mean “something happened that we may want to query later.” When teams page on every ERROR line in a log stream, they train responders to ignore the pager. The fix is not more alert rules. It is fewer, better rules with clear ownership.

Measuring the Difference

The difference between drained and sustainable is measurable. Teams that track these four numbers can see the rotation’s health without guessing:

• Pager load per engineer per week. Target below 25 percent. If the number is above 40 percent for two consecutive weeks, the rotation is draining people faster than it is protecting the service.
• Alert-to-action ratio. The share of pages that result in a documented action. A healthy ratio is above 80 percent. Below 50 percent means the team is paying attention to noise.
• Mean time to recovery for sev-2 and sev-3 incidents. Recovery time should trend down or stay flat as the team fixes root causes. If it trends up, the rotation is hiding unresolved complexity.
• Rotation coverage gaps per quarter. Zero is the target. One gap is a warning. Two gaps mean the schedule is not resilient to ordinary life events.

These metrics are not vanity numbers. They are the same signals a lean team can review in fifteen minutes during a weekly operations sync. The review does not need a dedicated SRE. It needs a shared spreadsheet or a simple dashboard in Grafana.

Alert Design Is the First Lever

Most drained rotations are drained by alert design, not by incident volume. A team that receives forty pages a week may have only four real incidents. The other thirty-six pages are threshold noise, duplicate alerts, or alerts that fire before a user-visible symptom exists.

The first lever is to page on symptoms, not causes. A symptom is something a user experiences: a slow checkout flow, a failed API call, a queue depth that will delay a batch job beyond its deadline. A cause is something an engineer investigates: high memory usage, a crashed pod, a saturated network link. Causes belong in dashboards and logs. Symptoms belong on pagers.

The second lever is to set alert thresholds from error budgets, not from intuition. An error budget is the amount of failure a service can absorb before users notice. For a service with a 99.9 percent monthly availability target, the budget is about 43 minutes of downtime. An alert should fire when the service is burning that budget faster than planned, not when a single request fails. Google’s SRE workbook describes this approach in detail and is a useful reference for teams that want to move from threshold guessing to budget-based alerting.

The third lever is to deduplicate and group alerts. A single failed node in a Kubernetes cluster can trigger alerts for pod readiness, node health, and ingress errors. The responder needs one page that says “node ip-10-0-4-7 is not ready; three pods affected,” not three pages that each describe a fragment of the same failure. Alertmanager grouping and inhibition rules handle this well for Prometheus-based stacks. PagerDuty and Opsgenie offer similar grouping for teams that use hosted incident response.

Runbooks Turn Pages into Learning

A runbook is a written procedure for responding to a specific alert. A good runbook answers four questions: What is the symptom? What is the first check? What is the fix? How do we verify recovery? A bad runbook says “investigate and resolve.”

For a lean team, runbooks do not need to be long. A runbook for a failed database backup might say:

1. Check the backup job status in AWS Backup or the scheduled cron output.
2. If the job failed due to permissions, compare the IAM role against the last known-good policy.
3. Rerun the backup manually and confirm the snapshot appears in the target region.
4. If the snapshot is missing after two attempts, page the secondary on-call and open a sev-2 incident.

That runbook is four lines. It is enough to prevent a 3 a.m. responder from guessing. It also creates a natural place to record what actually happened, which feeds the next incident review.

Gray Haven Lab’s earlier piece on writing the recovery checklist before you need it covers the structure of a useful runbook in more detail. The key point for on-call health is that a runbook is not documentation for its own sake. It is the difference between a page that costs twenty minutes and a page that costs two hours.

Incident Reviews Close the Loop

A sustainable rotation includes a lightweight incident review after every sev-2 or sev-3 page. The review does not need a formal postmortem template. It needs three questions: What happened? What did we learn? What will we change this week?

The “change this week” part matters. A review that produces a long list of future improvements is a drain. A review that produces one small change — a tighter alert threshold, a clearer runbook step, a removed duplicate alert — is a sustainability practice. The change should be small enough to complete before the next on-call shift starts.

For example, a team that received three pages for a flapping health check might decide to change the health check interval from 10 seconds to 30 seconds and add a 60-second stabilization window. That change takes an hour to implement and test. It removes a recurring page. The rotation gets quieter, and the team gets evidence that the review process works.

Access Hygiene Reduces After-Hours Surprises

On-call engineers need access to production systems, but that access must be scoped, temporary, and auditable. A drained rotation often pairs with standing AdministratorAccess policies for every engineer. A sustainable rotation uses short-lived credentials, break-glass roles, and a written record of who accessed what during an incident.

On AWS, a break-glass role with a 60-minute session duration and a CloudTrail log is a simple pattern. The on-call engineer assumes the role only when a page requires production access. The role’s permissions are scoped to the systems the runbook names. On GCP, the same pattern works with short-lived service account tokens and audit logs. For bare metal, SSH access via a bastion host with per-user keys and session logging serves the same purpose.

The benefit is not just security. It is predictability. When an on-call engineer knows exactly which systems they can touch and which they cannot, they spend less time guessing and less time waiting for someone else to grant access. The page resolves faster, and the rotation feels less like a free-for-all.

Monitoring That Supports Sleep

Monitoring for a sustainable rotation has one job: tell the on-call engineer when a user-facing service is broken or about to break. It does not need to tell them about every slow query, every retry, or every log line that contains the word “error.”

A useful starting point for a lean team is a small set of service-level indicators (SLIs) per service. For an HTTP API, the SLIs might be request latency at the 95th percentile, error rate as a share of total requests, and saturation of the primary dependency. For a batch job, the SLIs might be job completion time and data freshness. Each SLI gets one alert rule tied to an error budget. Everything else goes to dashboards and logs.

This approach reduces the number of pages without hiding real problems. It also makes the rotation easier to hand off. A new on-call engineer can learn five SLIs per service faster than they can learn fifty alert rules.

Staffing a Lean Rotation Without Burning People Out

A team of two to fifteen engineers cannot staff a follow-the-sun rotation across three time zones. It can, however, design a rotation that respects sleep and time off. The simplest pattern is a weekly primary with a named secondary. The primary handles pages during business hours and after hours. The secondary is available for escalations and for pages that require a second pair of hands. The secondary is not expected to be awake; they are expected to be reachable.

For a team of two, the rotation is inherently fragile. The mitigation is not to hire more people immediately. It is to reduce the number of things that can page. A two-person team should have fewer than ten alert rules total. If the team has more, the rotation will drain both engineers within a quarter.

For a team of eight to fifteen, the rotation can include a “shadow” week for engineers who are new to on-call. The shadow carries the pager with the primary but does not make changes alone. After two shadow weeks, the engineer takes a primary shift. This pattern builds coverage without throwing someone into a 3 a.m. incident on their first day.

Common Failure Patterns

Three failure patterns show up repeatedly in lean teams:

• The silent primary. One engineer handles most pages because they know the system best. The rotation looks staffed, but it is not. When that engineer takes a vacation, the team discovers that nobody else can run a recovery. The fix is to rotate primaries even when it feels slower.
• The alert snowball. A team adds a new alert for every incident. After six months, the pager fires constantly. The fix is to review alert rules monthly and remove any rule that has not paged for a real incident in the past 90 days.
• The reviewless incident. A page wakes someone, they fix the issue, and everyone moves on. The same issue pages again two weeks later. The fix is a five-minute review after every sev-2 or sev-3 page, with one concrete change assigned to a named owner.

Each of these patterns is easy to spot once the team tracks pager load and alert-to-action ratio. The hard part is not measurement. It is the willingness to remove alerts, rotate primaries, and write down what happened.

What to Change This Week

A team that wants to move from drained to sustainable can start with three changes:

1. List every alert rule and mark each as symptom or cause. Disable or demote the cause-based rules to dashboards. Keep only symptom-based rules on the pager.
2. Write a four-line runbook for the top three pages from the past month. Use the structure: symptom, first check, fix, verify. Link each runbook to the alert rule.
3. Track pager load and alert-to-action ratio for two weeks. Review the numbers in a 15-minute operations sync. Pick one noisy alert to fix before the next sync.

These changes do not require new tools. They require a shared document, a calendar, and a willingness to treat on-call as a system that can be improved rather than a tax that must be endured.

FAQ

What is a healthy pager load for a lean team?

A healthy pager load is below 25 percent per on-call engineer per week. That means the engineer spends less than a quarter of their on-call time responding to pages. Above 40 percent for two consecutive weeks is a signal that the rotation is draining people faster than it is protecting the service.

How many alert rules should a small team have?

A team of two to five engineers should aim for fewer than ten active alert rules. Each rule should page on a user-visible symptom tied to an error budget. If a rule has not paged for a real incident in 90 days, remove it or move it to a dashboard.

What is the difference between a runbook and a postmortem?

A runbook is a written procedure for responding to a specific alert. It answers what to check, what to fix, and how to verify recovery. A postmortem is a review of an incident after it is resolved. It answers what happened, what was learned, and what will change. A sustainable rotation uses both: runbooks to reduce response time, postmortems to reduce future pages.

How do we handle on-call when someone is on vacation?

Use a named secondary for every primary shift. The secondary is reachable for escalations but not expected to be awake. For a two-person team, reduce the alert surface before the vacation starts. Disable noisy alerts, document the top three runbooks, and agree on a break-glass escalation path.

What is the first metric to track if we track only one?

Track the alert-to-action ratio: the share of pages that result in a documented action. A healthy ratio is above 80 percent. Below 50 percent means the team is paying attention to noise. This single metric reveals whether the rotation is a learning loop or a drain.

The next step for teams that want to go deeper is to write the recovery checklist before the next incident. That practice turns the runbook from an afterthought into a pre-incident habit, and it is the natural follow-up to the changes described here.