Break-Glass Access Without a Backdoor: A Procedure for Lean On-Call Teams

by ·Comments Off on Break-Glass Access Without a Backdoor: A Procedure for Lean On-Call Teams

The failure a break-glass procedure exists for is rarely dramatic. More often it is a federation certificate that expired on a Saturday, an identity provider outage that coincides with a production incident, or an access policy change that locked every administrator out of the console. In each case the fix requires the same system that is broken. That circular dependency is the thing to design against.

AWS Well-Architected names this directly: an emergency access process lets authorized administrators reach cloud resources through alternate means when the centralized identity provider fails or federation configuration is modified. It also lists as an anti-pattern any emergency process that depends on the same systems as normal access, because failure of that system impairs both the primary and the emergency path. The same guidance notes that break-glass procedures can be considered a backdoor, and that the emergency account should only be accessed during emergencies.

For a team of two to fifteen engineers who are their own on-call rotation, the goal is not a cleverer admin account. It is a pre-provisioned, independently authenticated, dual-controlled, and rehearsed path that is visibly abnormal every time it is used.

Choose the pattern before you build the controls

NIST SP 800-53 Rev. 5, in its AC-2 discussion, draws a distinction that most small teams skip. Emergency accounts are established in response to crisis situations with a need for rapid activation, and their activation may bypass normal authorization. Accounts of last resort are different: they are infrequently used local or standing accounts that remain available and are not subject to automatic disabling or removal dates. The same discussion lists emergency accounts among the account types organizations may wish to prohibit because of increased risk.

Both patterns are legitimate. They fail differently and are audited differently. A crisis-created emergency account depends on someone or something being able to create or enable it during the outage, which may be unavailable in exactly the failure you are planning for. A pre-provisioned account of last resort exists all the time, which makes it a standing target but removes the creation dependency. Pick one deliberately and write down which one you built. Mixing the controls of one onto the other produces the worst of both: a standing account that is also expected to be created on demand, or a crisis account that is never rotated because nobody owns it.

For most lean teams the pre-provisioned account of last resort is the more honest choice, because it can be tested on a schedule and its custody can be rehearsed. The rest of this article assumes that choice and notes where the crisis-created pattern would differ.

The independence test

The single most useful review question is to walk the break-glass sign-in path and ask, at each step, what does this depend on? If any step touches the identity provider, the directory sync, a device-compliance service, a risk-scoring service, or a network location that the outage could remove, the path is not break-glass. It is a second copy of the primary path.

AWS Well-Architected recommends pre-creating the resources the emergency process needs, so the process does not depend on control plane APIs that may be unavailable during an emergency. In practice that means the emergency account, the roles, and the cross-account trust relationships exist before the incident, not during it. The same logic applies outside AWS: a GCP emergency grant that requires the identity provider to issue a token is not independent of an identity provider outage.

Keep the number of accounts small. Two is a common practice, not a rule: one may be unusable because a key is lost or a custodian is unavailable. More than a few becomes hard to monitor, and monitoring is the control that makes a standing account defensible.

Keep the exclusion list short and documented. The account is usually excluded from federation or directory synchronization, from device-compliance and network-location policies, and from automated inactivity disablement. It should not be excluded from logging, and it should not be excluded from multi-factor authentication if you can avoid it. A common approach is to register hardware security keys directly on the account so the second factor does not depend on an external MFA service.

Short-lived credentials where the platform supports them

Where the emergency path can issue temporary credentials, prefer them. AWS documentation on temporary security credentials states that they are short-term, can be configured to last from a few minutes to several hours, and after expiry AWS no longer recognizes them or allows API requests made with them. Because they have a limited lifetime, you do not have to update them or explicitly revoke them when they are no longer needed, and after expiry they cannot be reused.

Google Cloud IAM guidance makes a parallel recommendation: use conditional role bindings to let access expire automatically, and consider granting temporary elevated access. It also recommends granting roles to groups instead of individual users where possible, because group membership is easier to update than principals in allow policies.

This does not eliminate the need to rotate long-lived break-glass credentials. The AWS temporary-credentials page describes STS credentials, not the standing account of last resort. If your emergency path includes a long-lived password or key, rotation after use is still part of the procedure.

Custody that survives the outage it is meant for

Split knowledge means no single person knows the whole secret. Dual control means two people must act together to use it. The specific arrangement is secondary; the properties are what matter.

A workable arrangement for one account: one custodian holds part of the password, a second custodian holds the other part, and a hardware security key sits sealed in a separate location. No single custodian can sign in alone. The security key’s location is separate from both password holders. Any sign-in raises an alert.

The test to apply to any custody scheme is whether it still works during the outage you are planning for. A vault that requires the identity provider to unlock is not a custody solution for an identity provider outage. A secrets manager whose own break-glass path depends on the same directory is a second copy of the primary path.

Detection: every use is a signal

Because legitimate use is rare, alerting can be strict. Any successful or failed sign-in, any MFA method change, any password reset, and any role change on the emergency account should go straight to the people on call. AWS Well-Architected recommends generating detailed audit logs for all successful and failed attempts to gain emergency access, correlating activity with ongoing incidents, and alerting when the account is used during normal operations.

The alerting path must not depend on the system that might fail. If the identity provider’s own alerting is down during an outage, you still want to know that someone used the account. Route the alert through a channel that does not share the identity provider as a dependency, and confirm during rehearsal that it reached a human.

For teams running PostgreSQL, the emergency path should be tested against the actual authentication configuration rather than assumed. The pg_hba.conf file is read on start-up and when the main server process receives a SIGHUP; edits on an active system require signaling the postmaster with pg_ctl reload, pg_reload_conf(), or kill -HUP. The first record matching connection type, client address, database, and user is used, there is no fall-through or backup, and if no record matches, access is denied. A break-glass rule placed after a broader matching rule will never be reached. The test should confirm the specific rule is the one that matches, and that the file was reloaded after the edit.

Two authentication methods deserve explicit exclusion from any emergency rule. The trust method allows anyone who can connect to the server to log in as any PostgreSQL user without a password or other authentication. The password method sends the password in clear text over the network and should not be used on untrusted networks. Neither belongs in a break-glass path.

Rehearsal is where most procedures fail

An untested break-glass account is a guess. Credentials expire, keys get lost, policies change, and exclusions silently stop applying. AWS Well-Architected recommends testing emergency access processes periodically and as part of incident response simulations and disaster recovery tests.

A minimum viable rehearsal: two custodians retrieve the credentials and sign in. Confirm the account still holds the expected role. Confirm the exclusions still apply after recent policy changes. Confirm the alert fired and reached a human. Rotate the password, reseal the materials, and record the test. Quarterly is a common cadence; set your own and put it on the calendar.

If your recovery depends on restic repositories, rehearse the recovery tooling alongside the access path. Restic documentation notes that during a prune operation the repository is locked and backups cannot be completed, and advises planning pruning so it does not interfere with regular backup runs. It also advises running restic check after pruning to be alerted if the repository’s internal data structures are damaged. A prune that locks the repository during an incident, or a repository whose integrity was never checked after pruning, can turn an access success into a recovery failure. The --dry-run option of restic forget removes nothing and prints what actions would be performed, which makes it useful in a rehearsal.

Writing the recovery checklist before you need it is the companion exercise to the access rehearsal; see Write the Recovery Checklist Before You Need It for the recovery side of the same preparation.

After real use

After any real use, assume the credential has been exposed. Record who used it, when, from where, and why. Correlate with the incident ticket. Review every action taken with the account. Rotate the password and, if a key left its sealed storage, consider re-registering it. Reseal the custody materials and update the custody log. AWS Well-Architected recommends automatically rotating emergency access credentials and notifying the relevant teams upon returning to normal operations.

Then fix the root cause that required the account, so the next outage does not need it. A break-glass procedure that is used repeatedly for the same failure is a signal that the primary path has a defect worth fixing.

Failure modes to design out

Federated break-glass accounts are the most common mistake. An account that authenticates through the directory whose failure it is meant to cover is not an emergency path. Shared vaults that every administrator can open turn the account into a shared admin account. Excluding the account from MFA and from logging to keep it simple removes the controls that make it defensible. Never testing means the first real use discovers an expired password or a key nobody can find.

Routine use is the failure mode that turns break-glass into a backdoor. AWS Well-Architected lists as an anti-pattern emergency access processes used in non-emergency situations, for example when users find it easier to make changes directly than to submit them through a pipeline. If the account is convenient, it will be used for convenience. The design should make routine use visibly abnormal: every use alerts, every use is reviewed, and every use is recorded against an incident.

A one-page runbook skeleton

Fill this in and keep it where the on-call engineer can find it without the identity provider.

  • Trigger conditions. Name the specific failures that count as an emergency: identity provider unavailable, federation certificate expired, access policy locked out all administrators. Name the situations that do not count.
  • Who may initiate. Name the roles, not the people, and name the approvers and the escalation path if the primary approver is unavailable.
  • Custody map. Who holds which part, where the security key is sealed, and how the two custodians reach each other during an outage.
  • Alert destination. The channel that receives every sign-in attempt, and the independent path used to confirm it arrived.
  • Test cadence. The scheduled rehearsal date, the two custodians, and the record location.
  • Post-use checklist. Record, correlate, review, rotate, reseal, fix the root cause.

Questions that come up

Is a pre-provisioned account of last resort the same as an emergency account?

No. NIST SP 800-53 Rev. 5 AC-2 discussion distinguishes emergency accounts, created or enabled during a crisis and subject to automatic removal or disabling after a defined period, from accounts of last resort, which remain available and are not subject to automatic disabling or removal dates. Both are legitimate; the controls differ. Decide which one you are building and document it.

Should the break-glass account require MFA?

Yes, where you can. The risk is a standing, high-value secret. Registering hardware security keys directly on the account keeps the second factor independent of any external MFA service. Excluding the account from MFA to keep it simple removes the control that makes a standing account defensible.

How often should we test?

Quarterly is a common practice, not a requirement. The important properties are that two custodians participate, that the alert is confirmed to reach a human, and that the test is recorded. AWS Well-Architected recommends periodic testing and testing as part of incident response simulations and disaster recovery tests.

What if the emergency account is used for routine work?

Then it is no longer break-glass. AWS Well-Architected lists routine use as an anti-pattern. Every use should alert and be reviewed against an incident ticket. If the account is convenient, the design has failed.

Does a temporary credential remove the need to rotate?

Only for the temporary credential itself. AWS documentation states that temporary security credentials expire and cannot be reused after expiry, so you do not have to update or explicitly revoke them. A long-lived password or key in the custody scheme still needs rotation after use.

Sources