A controlled failover test can reveal what your DNS setup actually does. The first 60 seconds after stopping the endpoint are not a single event. They are a sequence of independent caches, resolvers, health checks, and client retry policies, each with its own timer. For a lean team running production without a dedicated SRE department, the goal of a DNS failover rehearsal is not to prove the failover works. It is to measure how long the failover takes, which clients recover first, and which clients keep talking to a dead address.

This article is a proposed observation plan, not a report of a test Gray Haven Lab performed: choose a low-traffic window, stop a test endpoint under an approved change plan, and observe the recovery curve for at least 60 seconds. We will cover what DNS failover means in practice, why TTL is not a guarantee, how health checks interact with authoritative DNS, and what to record so the next rehearsal is faster. The companion piece, Write the Recovery Checklist Before You Need It, covers the pre-work that makes this rehearsal safe.

What DNS failover actually is

DNS failover is the practice of changing the answer an authoritative nameserver returns when a health check marks an endpoint unhealthy. It is not a load balancer. It is not a connection drain. It is a control-plane change that propagates through a distributed cache hierarchy at a speed you do not control.

Adjacent concepts matter here: record TTL, negative caching, resolver prefetch, client connection pooling, and health-check interval plus failure threshold. A DNS failover rehearsal exercises all of them at once. That is why the 60-second window is useful. Treat it as an initial observation window, not a promised recovery time or a guarantee of limited impact.

For a small team that runs its own on-call rotation, DNS failover may be a practical cross-region or cross-provider mechanism. It is also the mechanism most likely to be assumed rather than measured. The rehearsal exists to replace the assumption with a number.

The 60-second timeline, second by second

The following timeline is an observation template, not measured traffic or a statement of any provider’s defaults. Record your actual health-check interval, failure threshold, record TTL and client retry behavior before the test. Use the time bands below to organize observations, not to predict when your provider will switch an answer.

T+0s: you kill the healthy endpoint

At T+0, the endpoint stops answering. Stopping a process may produce an explicit connection failure; dropping network traffic may instead leave clients waiting for a timeout. Record the behavior you actually observe. Record which one you did. A process kill and a path blackhole produce different health-check behavior.

T+0s to T+30s: health checks accumulate failures

Watch the health-check results rather than assuming an interval multiplied by a failure threshold predicts the transition. Record the first failed check, the time the provider marks the endpoint unhealthy, and the first observed answer containing the failover target. Different observers or clients may see those events at different times.

During this window, clients are still receiving the healthy address. Some of them are failing. Some of them are retrying. Your error rate is already rising before DNS has changed anything.

T+30s to T+45s: authoritative DNS changes

Once the health check marks the endpoint unhealthy, the authoritative nameserver begins returning the failover target. This is a control-plane event. It does not push to resolvers. It waits for them to ask again.

If your record TTL is 60 seconds, a resolver that cached the healthy answer at T-1s will keep serving it until T+59s. If your TTL is 300 seconds, that resolver keeps serving the dead address until T+299s. This is the second surprise: the authoritative change is not the client-visible change.

T+45s to T+60s: the first wave of clients moves

Clients with short-lived resolvers, no caching, or aggressive retry logic move first. Clients behind corporate resolvers, ISP resolvers, or long-lived connection pools move later. There is no universal percentage of requests that will move by T+60s. Record the share on the failover target at T+60s, T+120s, and T+300s; the measured curve is the number worth keeping.

If you use a global accelerator or anycast front door, the timeline compresses because the client is not waiting on recursive resolvers. If you use plain DNS with a 300-second TTL, the timeline stretches. Neither is wrong. Both need to be known.

Why TTL is not a guarantee

TTL is a hint, not a contract. Resolver behavior and client-side caches can make the visible change differ from the record’s nominal TTL. Negative caching for NXDOMAIN and NODATA responses is governed separately by the SOA minimum TTL, whose actual configured value should be checked. If your failover depends on a fast negative-cache expiry, check the SOA minimum before the rehearsal, not during it.

Client-side DNS caching adds another layer. Runtime DNS cache settings and long-lived connection pools can extend the time before a client uses a new address. Inspect the effective settings of each important client rather than assuming its behavior. A DNS change does not reach a process that already has an open connection. This is why a rehearsal should include at least one long-lived client, not just curl loops.

Primary sources worth reading before you set TTLs: the IETF RFC 1035 definition of TTL and the RFC 2308 treatment of negative caching. Both are short and both explain why your 30-second TTL may not behave like 30 seconds.

Health checks are part of the failover, not a precondition

A common rehearsal mistake is to treat the health check as a binary gate. In practice, the health check is a timer with its own failure modes. A health check that depends on a single TCP port will mark unhealthy when the port closes, even if the application is still serving on another port. A health check that depends on an HTTP 200 will mark unhealthy during a slow database query, even if the endpoint is otherwise fine.

Before the rehearsal, write down the health-check interval, the failure threshold, the success threshold, and the protocol. After the rehearsal, compare the observed unhealthy transition time to the calculated one. If they differ by more than one interval, the health check is doing something you did not expect.

For teams using PagerDuty escalation, the rehearsal should also confirm that the failover event does or does not page. A DNS failover that pages the on-call engineer at 03:00 is a different operational decision than one that does not. Decide which you want before the rehearsal, not after.

What to record during the 60 seconds

The rehearsal produces a dataset, not a pass/fail. Record at least these fields:

• Kill method: process stop, port close, or network blackhole.
• Health-check interval, failure threshold, and protocol.
• Record TTL and SOA minimum TTL at the time of the rehearsal.
• Time to authoritative change, measured from the provider’s API or console.
• Time to first client request on the failover target, measured from access logs.
• Percentage of request volume on the failover target at T+60s, T+120s, and T+300s.
• Error rate on the killed endpoint during the window.
• Whether the event paged, and at what severity.

As a hypothetical diagnostic example, suppose a small share of requests still reaches the stopped endpoint after the record TTL has passed. Compare those requests by client and connection age. A long-lived connection pool or runtime cache is one possible cause; verify it in logs before changing TTL or retry settings.

Tradeoffs: shorter TTLs versus resolver load

Shortening TTLs reduces the failover window but increases query volume against your authoritative nameservers. For a small team, that tradeoff is usually acceptable for the specific records that participate in failover, and not acceptable for every record in the zone. Scope the change.

Another tradeoff: aggressive health checks detect failure faster but generate more false positives during transient network blips. A 10-second interval with a 1-failure threshold will mark unhealthy on a single dropped packet. A 10-second interval with a 3-failure threshold tolerates two. Neither is universally correct. The rehearsal tells you which one matches your error budget.

If you use a managed failover product, read the provider’s documented behavior for health-check evaluation and failover timing. AWS documents Route 53 health-check behavior in its DNS failover documentation. Cloudflare documents its load-balancing health checks in its health check documentation. These are the primary sources for the timers you are about to measure.

Pre-mortem questions before the next rehearsal

A pre-mortem is cheaper than a postmortem. Before the next rehearsal, answer these:

• What is the worst thing that happens if the failover target is also unhealthy?
• Which clients are known to cache DNS beyond TTL?
• Does the failover target have capacity for the full production load, or only a fraction?
• Who is authorized to trigger the rehearsal, and who is authorized to abort it?
• What is the rollback path if the failover target degrades during the window?

These questions belong in the recovery checklist, not in the incident channel. The recovery checklist article covers how to structure them so they are usable under pressure.

Access hygiene around the rehearsal

A DNS failover rehearsal touches production DNS. That means it touches credentials. Before the rehearsal, confirm that the engineer running it has the minimum permissions required, that the break-glass path is documented, and that the change is logged in the same place as other production changes. After the rehearsal, confirm that any temporary credentials or elevated roles are revoked.

For teams with onboarding and offboarding processes that are still manual, the rehearsal is a good moment to test whether a departed engineer’s credentials would still work. If they would, that is a finding. Record it as a finding, not as a failure of the rehearsal.

Monitoring and failure rehearsal

Synthetic checks are one way to observe the client-visible failover curve. A synthetic check from two or three external regions, polling at a documented interval, can give you a time series showing when the failover target started answering. Pair it with a growth-rate alert on the failover target’s request rate, so you notice if the failover target is absorbing traffic faster than expected.

If you use PagerDuty escalation policies, decide whether the rehearsal should page. If it should not, suppress the alert for the window and record the suppression. If it should, treat the page as a real page and follow the escalation path. Both are valid. The invalid option is not deciding.

FAQ

How long should a DNS failover rehearsal last?

Start with a 60-second timeline, but keep observing until the authoritative answer and the important client paths have actually changed. With a 300-second TTL or a slow health check, that may require ten minutes or more. Record the stopping rule before the test.

Should I lower TTL before the rehearsal?

Lower it at least one old-TTL period before the rehearsal, so resolvers have had a chance to pick up the new value. If you lower it at T-0, many resolvers will still be serving the old TTL.

What if the failover target cannot handle full load?

Then the rehearsal should measure partial failover, not full failover. Record the capacity limit and the resulting error rate. A failover that works at 40% load and fails at 100% is a known limitation, not a surprise.

Does a global accelerator remove the DNS failover window?

It compresses it. Anycast and global accelerators move traffic at the network layer, so clients do not wait on recursive resolvers. The health-check and control-plane timers still apply, but the resolver-cache portion of the window is largely removed.

How often should we rehearse?

Choose a cadence that matches your change rate and risk. If the failover path changes, rehearse after the change. If the on-call rotation changes, rehearse after the change. Budget the test and its rollback as a real production change.

What to do next

Pick one low-traffic service with a DNS failover path. Write down the health-check interval, failure threshold, record TTL, and SOA minimum. Get an approved change window and rollback plan. Stop the chosen endpoint, record the timeline for as long as the configured timers require, and compare observed numbers with the plan. Then update the recovery checklist with what you learned.

The follow-up topic worth writing next is the client-side half of this problem: how long-lived connection pools and runtime DNS caches can extend the failover window beyond the DNS record’s TTL. Measuring that behavior helps explain whether a real failover feels like a brief interruption or a longer outage.