A third-party API starts returning 429s. Your queue backs up, your retries pile on, and by the time someone notices, the on-call engineer is staring at a dashboard that says “rate limited” and a vendor status page that says “all systems operational.” The incident ends. Now you have to write it up.

The temptation is to file a short postmortem that says “vendor throttled us, we backed off, resolved.” That document will not help the next on-call engineer, and it will not prevent the next incident. The useful postmortem for a rate-limit incident records three things the team can actually verify: what the dependency did, what your own client did, and what you changed afterward. Everything else is narrative.

This is written for teams of two to fifteen engineers who run their own on-call rotation and have no dedicated reliability staff. The structure below assumes you already write postmortems; if you are still deciding what belongs in one, Google’s Postmortem Culture chapter defines the baseline: a postmortem is “a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.” A rate-limit incident fits that definition, but each section needs different evidence than a code-defect incident does.

Start with impact and timeline, not with the vendor

Record what users experienced before you record why. Which internal services degraded, for how long, and what the user-visible symptom was. A batch pipeline that fell two hours behind is a different incident from a checkout flow that returned errors for eleven minutes, even if both trace to the same 429.

The timeline needs two specific timestamps that teams often skip: when the first rate-limit response appeared, and when a human first knew about it. The gap between those two is your detection latency, and it is usually the most actionable number in the document. If the first 429 arrived at 02:14 and the first page fired at 02:51, the postmortem should say so plainly, because that thirty-seven-minute gap is a monitoring problem you can fix independently of the vendor.

Also record how the incident was detected. An alert on error rate, a synthetic check, a customer report, and a colleague noticing a stale dashboard are four different detection paths with four different remediation stories. Name the one that happened.

Log the dependency’s observed behavior verbatim

This is the section that saves the next on-call engineer the most time, and it is the one most often written from memory. Do not write it from memory. Paste the actual response.

For each rate-limit response you captured, record:

• The HTTP status code and the full response body or error payload, verbatim.
• Any rate-limit or retry headers the vendor returned, with their values.
• The request ID or trace ID from the response, if the vendor provides one, so a support ticket can reference it.
• The timestamp of the response, in UTC, and the endpoint or method that triggered it.

The payload matters because “429” is not a complete description. Google’s API design guidance, AIP-193, shows a 429 response carrying a RESOURCE_EXHAUSTED status and a machine-readable ErrorInfo block with a reason and domain. A vendor that follows that pattern gives your client something to branch on. A vendor that returns a bare 429 with an HTML error page gives you nothing, and that difference belongs in the postmortem because it shapes what your retry logic can safely do.

Then record what the vendor’s public documentation says about throttling, and label it as documentation rather than observation. AWS’s API Gateway documentation, for example, states that throttles and quotas “are applied on a best-effort basis and should be thought of as targets rather than guaranteed request ceilings,” and that clients may receive 429 responses they can resubmit “in a way that is rate limiting.” That is a documented behavior, not something you observed in your incident. Keep the two categories separate in the write-up. If the vendor documents a token-bucket model, say so and cite the page. If you are inferring the model from response patterns, say that instead.

Record your own request pattern with numbers

The vendor’s behavior is half the story. Your client’s behavior is the half you can change, and it needs numbers.

Capture the request rate your service was sending at the time of the incident, and the rate in the hour before. Note whether the traffic was steady or bursty, and what caused any change: a scheduled batch job, a retry storm, a traffic spike, a deploy that changed concurrency. If you have per-endpoint or per-tenant breakdowns, include them, because a single noisy tenant can consume a shared quota and the fix may be per-tenant limiting rather than a global change.

Then document the retry logic as it actually existed, not as you intended it to work. Specifically:

• Were retries bounded, and at what count or duration?
• Did retries use exponential backoff, and did they include jitter?
• Did the client honor any Retry-After header the vendor sent?
• Did retries run concurrently across workers, so that each worker’s backoff was independent and the aggregate rate stayed high?
• Did the retry logic distinguish a 429 from a 500, or did it treat all failures the same way?

Retries deserve their own subsection because they can turn a rate-limit incident into a self-inflicted one. If your client retried immediately on 429, the postmortem should say so and should show the request rate before and after retries began. That is an observation about your system, not a judgment about the engineer who wrote the retry loop.

Separate the trigger from the contributing conditions

A rate-limit incident rarely has a single root cause, and forcing one into the document produces a misleading record. Write down the trigger and the contributing conditions as separate lists.

The trigger is the event that started the incident: a traffic spike, a batch job that doubled the request rate, a vendor-side quota change, a retry storm. The contributing conditions are the properties of your system that let the trigger become an incident: no client-side limiter, no alert on 429 rate, no documented vendor quota in the runbook, no circuit breaker between your service and the dependency.

This split is what keeps the postmortem useful. “Vendor rate-limited us” is a trigger with no contributing conditions attached, and it produces no action items. “A batch job doubled our request rate; we had no client-side limiter and no alert on 429 rate” produces two fixes the team controls.

It also keeps the document blameless in the specific way that matters here. Google’s guidance is that a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.” A vendor is a team. The postmortem should describe what the vendor’s API did, with evidence, and should not characterize the vendor’s motives or competence. The same rule applies to the engineer who wrote the retry loop two years ago.

Label inferences about the vendor explicitly

You can observe the vendor’s responses. You cannot observe the vendor’s internal throttling state, its quota accounting, or why a particular request was rejected. When the postmortem needs to explain vendor behavior beyond what the response and documentation show, mark it as an inference.

A workable convention: put observed facts in the main text with a timestamp or a pasted payload, and put inferences in a clearly headed subsection such as “What we believe happened on the vendor side.” That subsection can say “we infer the quota is shared across our three API keys because all three began returning 429s within the same minute” without asserting it as fact. If a support ticket later confirms or contradicts the inference, update the postmortem and note the update date.

Record vendor communication and support tickets

If the team opened a support ticket, record the ticket number, the time it was opened, the time of the first response, and the resolution or current status. If the vendor’s status page was updated during the incident, note the timestamp of the update and what it said. If the team requested a quota increase, record the request date and the outcome.

This section is short but it prevents a common failure: six weeks later, nobody remembers whether the quota increase was approved, and the next incident repeats the discovery. If your contract includes service credits or SLA terms that the incident might qualify for, note the relevant clause and whether the team filed a claim. Do not estimate the credit amount in the postmortem; record the clause and the filing status, and let finance or the account owner handle the number.

Write action items with one owner and a tracking reference each

The action items are where the postmortem either changes something or becomes a document nobody reads again. Google’s SRE Workbook is direct about the failure mode: “Action items without clear owners are less likely to be resolved. It’s better to have a single owner and multiple collaborators.” The same chapter criticizes action items that use vague verbs like “improve” or “make better,” and notes that items tagged with equal priority give no guidance on what to do first.

For a rate-limit incident, split the action items into two groups.

Preventative items change the system so the incident is less likely or less severe:

• Add a client-side rate limiter with a configured ceiling below the vendor’s documented quota.
• Add exponential backoff with jitter to the retry path, and honor Retry-After when present.
• Add an alert on the 429 rate, or on the ratio of 429s to total requests, with a threshold tied to your observed baseline.
• Add a circuit breaker that stops sending requests when the 429 rate crosses a threshold, so the dependency can recover.
• Document the vendor’s quota and burst behavior in the runbook, with a link to the vendor’s documentation page.

Mitigative items shorten the next incident:

• Add a runbook entry describing how to identify a rate-limit incident, how to reduce request rate quickly, and how to open a vendor ticket.
• Add a dashboard panel showing request rate, 429 rate, and retry count side by side.
• Record the vendor’s support escalation path and the account identifier needed to open a ticket.

Each item gets one owner, a priority, and a tracking reference such as an issue number. If your team uses a postmortem template, the template should have columns for all three, because a table with an empty owner column is easier to notice than a paragraph that omits one.

What to leave out

Three things do not belong in this postmortem.

First, animated language. The SRE Workbook’s critique of a bad postmortem notes that “animated language and dramatic descriptions of events distract from the key message and erode psychological safety.” “The vendor completely failed us” and “the retry loop was obviously broken” are both animated, and both make the document harder to act on.

Second, claims about the vendor’s internal behavior that you cannot support. If the vendor’s documentation does not state a quota, do not write a number as if it were documented. Write what you observed and label the rest as inference.

Third, a single root cause when the evidence supports several contributing conditions. The trigger-plus-conditions structure is more accurate and more actionable than a one-line cause statement.

A constructed example

The following is an illustration, not a case record. A service calls a vendor API at roughly 50 requests per second with no client-side limiter. The vendor’s documentation states a steady-state limit of 40 requests per second and a burst of 100. A nightly batch job starts and doubles the request rate to about 100 per second. The vendor returns 429s with a JSON body containing a Retry-After header of 2 seconds. The client’s retry loop ignores the header and retries immediately, pushing the aggregate rate higher. The first 429 appears at 02:14; the on-call engineer is paged at 02:51 by an alert on queue depth, not on 429 rate.

The postmortem for that incident would record the pasted 429 payload and the Retry-After value, the request rate before and after the batch job started, the retry behavior including the ignored header, the detection gap of thirty-seven minutes, and action items for a client-side limiter, jittered backoff that honors Retry-After, and an alert on 429 rate. It would note that the vendor’s documented burst of 100 was exceeded and that the team inferred the batch job was the trigger, with the batch job’s start time as evidence. It would not say the vendor “throttled us unfairly” or that the engineer who wrote the retry loop “should have known.”

A checklist to paste into your template

The next time a third-party rate limit causes an incident, the postmortem should answer these questions:

• What did users experience, for how long, and which internal services were affected?
• When did the first rate-limit response appear, and when did a human first know?
• What was the exact status code, response body, and any retry headers, pasted verbatim?
• What does the vendor’s public documentation say about throttling, and what did we infer beyond it?
• What was our request rate before and during the incident, and what changed it?
• What did our retry logic actually do, including whether it honored Retry-After and whether it used jitter?
• What was the trigger, and what were the contributing conditions?
• What vendor communication happened, with ticket numbers and timestamps?
• Which action items are preventative, which are mitigative, and who owns each one?

If your team is still building the habit of writing recovery procedures down before an incident, the companion piece on writing the recovery checklist before you need it covers the runbook side of the same problem. The postmortem records what happened; the runbook is what the next on-call engineer reads at 02:14.