A DNS provider outage drill that changes nameservers is not a rehearsal. It is the incident, performed on a Tuesday because the calendar said so. For a lean team that is its own on-call rotation, the useful rehearsal is narrower and safer: exercise what you can observe, decide, and communicate while your authoritative provider is unreachable, and leave the delegation records alone.
This article describes that drill. It is a read-only exercise plus a tabletop. It does not validate failover, and it should not be reported as if it did.
Scope the drill before you schedule it
Two questions belong in a DNS provider outage drill, and neither requires touching production state:
- Observation. When the provider’s control plane is unavailable, what do our checks actually show? Which queries fail, which are answered from cache, and which look healthy while being stale?
- Decision. Who declares the outage, what evidence do they require, what will they refuse to do under pressure, and what do they escalate?
Changing NS records at the registrar is a separate activity with its own risk profile. AWS’s migration guide is explicit about the cost of getting it wrong: if you do not lower the NS record TTL, a domain could be unavailable on the internet for up to two days if something goes wrong during a migration (AWS, Making Route 53 the DNS service for a domain that’s in use). That is not a step to rehearse casually on a live zone. The drill described here deliberately excludes it.
Understand what you are simulating
Route 53’s documentation separates a control plane, which handles management operations such as creating and updating resources, from a data plane that provides core functionality (AWS, Resilience in Amazon Route 53). AWS recommends data-plane mechanisms such as health checks and ARC routing control for failover during disaster recovery, rather than relying on control-plane operations (AWS, Best practices for Amazon Route 53 DNS).
That distinction gives the drill its scenario. The realistic provider outage for most lean teams is not “every authoritative server stops answering.” It is “the console and API are unavailable or degraded, and we cannot make changes for a while.” Existing records keep being served from the provider’s data plane and from resolver caches. Your team is blind to the control plane and must decide whether to wait, escalate, or invoke a break-glass path.
Write the scenario down in those terms. Label it hypothetical. A scenario is not an incident report, and the drill artifact should say so.
Build the observation kit first
The drill is only as good as the evidence the on-call engineer can gather in the first ten minutes. Prepare the commands before the window, and have the on-call run them against live production records in read-only mode.
Compare authoritative and recursive answers
Query the provider’s authoritative nameservers directly, then query public resolvers, then query your own resolver. A practical troubleshooting guide recommends exactly this split: query multiple resolvers and authoritative servers directly rather than testing with only one resolver, and use dig +trace or direct queries against the authoritative nameservers to distinguish authoritative from recursive answers (Azion, DNS troubleshooting).
In a provider control-plane outage, the authoritative answers should still be correct. If they are, and a resolver returns old data, the guide’s rule applies: the issue is caching or resolver-related, not authoritative (Azion). That distinction is the single most useful thing the on-call can establish during the drill, because it determines whether the team waits or escalates.
Capture TTL and negative caching behavior
Resolvers cache responses for the TTL, and some caches also retain negative answers for a period (Azion). During the drill, record the TTLs on the records that matter — apex A/AAAA, the NS set, MX, and any health-checked failover records — and note which resolvers return which remaining TTL. This is observation, not measurement of failover time.
RFC 1034 explains why stale answers are expected rather than anomalous: access to information is more critical than instantaneous updates or guarantees of consistency, and when updates are unavailable due to network or host failure, the usual course is to believe old information while continuing efforts to update it (RFC 1034). A drill that treats cached answers as a bug will produce the wrong decision.
Trace the delegation chain
Run dig +trace for one or two critical names and record where the chain resolves from cache and where it reaches the provider. This exercises the team’s ability to answer the question “is the delegation intact?” without changing it. It also surfaces the parent-zone TTL issue that AWS documents: some full resolvers might cache the TTL of the NS record of the parent authoritative server, so that TTL must also be reduced during a real migration (AWS). The drill does not change that TTL; it records what it currently is and who owns it.
Timestamp everything
Every command in the kit should be run with a timestamp in the drill log. The timestamps are the raw material for the decision record, and they are the only honest basis for any later claim about how long the team took to reach a conclusion.
Run the drill as a tabletop plus read-only probing
Pick a 60–90 minute window. Announce it to the team as a drill, not an incident. The on-call engineer walks the existing runbook against live read-only queries while a second person plays the provider’s status page and the escalation contact.
A workable sequence:
- Inject the scenario. Read the hypothetical aloud: “The provider console and API have been unreachable for 20 minutes. Status page says investigating. No ETA.”
- Observe. The on-call runs the observation kit and writes down what each check shows, with timestamps.
- Decide. The on-call states, out loud, what they would do next and what evidence they used. The second person records the decision and the evidence.
- Escalate. The on-call states who they would contact, in what order, and what they would say. No actual pages are sent unless the team has agreed to a separate escalation test.
- Stop. At the end of the window, the drill ends. No production changes were made, and the log says so.
The runbook itself should already exist. If it does not, write the recovery checklist before the drill, not during it; the Recovery Checklist article covers that preparation step.
Measure the declaration gap in miniature
The most valuable artifact from this drill is a small, honest record of the gap between “we would declare this handled” and “we would have evidence that traffic moved.” A practitioner account of a real failover describes the pattern: the DNS record changed in seconds, but traffic moved 18 minutes later, and the author argues that failover is complete only when traffic distribution changes, not when the record changes (Rack2Cloud, DNS failover testing). The same account argues that a test confirming the record updated and the health check passed has validated the protection plane, not whether traffic actually moved (Rack2Cloud).
That account is one anecdote, not a benchmark. Do not import its 18 minutes into your own RTO model. Instead, use the distinction it draws as a checklist item: during the drill, ask the on-call to state separately (a) when they would declare the provider issue resolved and (b) what evidence they would require before believing traffic had returned to normal. Record both answers and the gap between the evidence they named and the evidence they actually have access to.
If the team cannot measure traffic distribution at all, the drill should record that as an open question and schedule the measurement separately. A read-only drill does not validate failover capability. It validates observation and decision. Saying otherwise in the postmortem is how a team convinces itself it has tested something it has not.
Write the decision record
The drill’s output is a short decision record, not a narrative. It should answer, in plain sentences:
- Who declares a DNS provider outage, and what evidence do they require before declaring it?
- What will the team not do under pressure — for example, change NS records at the registrar without a second approver, or edit records repeatedly during the incident? The troubleshooting guide’s advice is to stabilize, lower TTL before planned migrations, and change once, rather than changing records repeatedly during an incident (Azion).
- What is the escalation path, and who owns the provider relationship?
- Which signals were missing during the drill, and which of them can be added before the next one?
- What remains hypothetical, and what would need to be measured to turn it into a fact?
File the record where the team keeps runbooks and decision records, and link it from the runbook. The next drill should start by reading it.
Keep the drill honest
Three rules keep a no-change DNS drill from turning into folklore:
Label hypotheticals as hypotheticals. The scenario is invented. The commands were real. The timestamps are real. The conclusions about what the team would do are decisions, not observations. Write them that way.
Do not claim measured RTOs you did not measure. A drill that never moved traffic cannot report a recovery time. It can report how long the team took to reach a decision, which is a different and still useful number.
Do not present the drill as a failover test. The distinction between record change and traffic movement is the point. A drill that blurs it teaches the team the wrong lesson at the worst possible moment.
What this drill does not cover
It does not exercise registrar access, DNSSEC key changes, or secondary-provider cutover. Those involve production state changes and need their own preparation, approvals, and rollback plans. AWS’s migration guide is the right starting point for the cutover itself, including its warning that DNSSEC signing cannot be enabled across two providers at the same time and that the DS record must be removed from the parent zone before migration (AWS). None of that belongs in a read-only rehearsal.
What the drill does cover is the part most lean teams skip: the ten minutes before anyone touches anything, when the on-call has to decide whether the problem is authoritative, recursive, or somewhere in between, and whether to wait or escalate. That decision is worth rehearsing. The nameservers can stay where they are.
Questions teams ask before running this
Can we run this during business hours?
Yes. The drill makes no production changes. The main constraint is that the on-call should not be simultaneously handling a real incident, so pick a window when the rotation is quiet and announce it in advance.
Do we need a second DNS provider to run it?
No. The drill is about observation and decision during a control-plane outage. A secondary provider is a separate architectural decision with its own rehearsal requirements.
How do we know the drill was useful?
If the decision record names at least one missing signal, one unclear owner, or one runbook step that could not be followed as written, the drill did its job. A drill that produces no gaps usually means the team did not probe hard enough.
Should we lower TTLs before the drill?
No. Lowering TTLs is a production change and belongs to a planned migration, not a read-only rehearsal. AWS recommends temporarily shortening TTLs when making changes to critical DNS entries so changes can be observed and rolled back quickly (AWS), but that advice applies to the change window, not the drill.