The 48-Hour Offboarding Revocation Checklist: Pulling Access Without Breaking the On-Call Rotation

by ·Comments Off on The 48-Hour Offboarding Revocation Checklist: Pulling Access Without Breaking the On-Call Rotation

Offboarding is a reliability event disguised as an HR event. When an engineer leaves a 2–15 person team, the same person who held production access, carried a pager, and knew which runbook to open at 03:00 is now a set of credentials that must be revoked. The failure mode is not that revocation is forgotten; it is that revocation is done in the wrong order, and the on-call rotation loses a responder, a break-glass path, or a backup credential at the moment it is needed.

This checklist is written for teams that are their own SRE function. It assumes AWS, GCP, or bare metal, PostgreSQL with WAL archiving, restic or pgBackRest for backups, and PagerDuty or an equivalent escalation tool. It is a sequence, not a policy document. The goal is to remove a person’s access within 48 hours while keeping the rotation intact.

Why 48 hours, and why order matters

NIST SP 800-53 Rev. 5, the control catalog for federal information systems, includes an Access Control family covering account management. The operational reading for a lean team is that revocation must be a defined procedure with an owner, not an ad hoc action. The 48-hour window is a team-level target, not a NIST requirement. Treat it as a service-level objective for your own process.

Order matters because access is layered. A departing engineer may hold a long-lived IAM user with an access key, a federated identity through an IdP, a PagerDuty user with a schedule override, a database role with a password in a password manager, and an SSH key on a bastion. Revoking the IAM user first can break a script that the rotation depends on. Revoking the PagerDuty user first can leave a gap in the escalation policy. The sequence below removes human access first, then service access, then shared secrets, then the pager identity.

Phase 1: Freeze and inventory (0–4 hours)

Before removing anything, capture what exists. This is the step that prevents the 03:00 surprise.

  • List all IAM users, roles, and access keys associated with the person in AWS. In AWS, temporary credentials issued through AWS STS are short-lived and expire on their own; long-lived IAM user access keys do not. The AWS IAM documentation states that temporary credentials can be configured to last from a few minutes to several hours and that after they expire, AWS no longer recognizes them. If the departing engineer’s access is federated through an IdP, the long-lived key inventory may be smaller than expected.
  • List all GCP principals. Google Cloud IAM grants roles to principals on resources, and allow policies are inherited down the resource hierarchy. A role granted at the organization or folder level applies to every descendant resource. Revoking a project-level binding does not remove an inherited organization-level binding. The Google Cloud IAM overview documents this inheritance explicitly: to understand who can access a resource, you must view the resource’s allow policy and its ancestors’ allow policies.
  • List PagerDuty schedule memberships, escalation policy steps, and any user-level notification rules. The PagerDuty integrations documentation describes the platform as connecting tools and data to incident management; the operational point is that a user is a first-class object in schedules and escalation policies, not just an email address.
  • List database roles and authentication methods. For PostgreSQL, check pg_hba.conf for any trust or md5 entries tied to a specific host or user, and check for roles with SUPERUSER or REPLICATION attributes.
  • List backup credentials. If restic repositories are accessed with a password stored in a shared vault, note who knows it. If pgBackRest uses an SSH key or a cloud identity, note which one.

Write the inventory to a ticket or a decision record. The inventory is the artifact that makes the rest of the checklist auditable.

Phase 2: Remove human access (4–24 hours)

Human access is the interactive path: console logins, CLI sessions, SSH sessions, and database sessions. Remove it before touching service accounts.

AWS

If the engineer has an IAM user, deactivate the access keys and delete the login profile. If the engineer accesses AWS through a federated identity, disable the identity in the IdP. The AWS documentation notes that temporary credentials are the basis for roles and identity federation and that they do not have to be explicitly revoked when no longer needed because they expire. That property is useful during offboarding: a federated session that is already issued will expire on its own, but the IdP must be disabled so new sessions cannot be issued.

Check for any IAM roles with a trust policy that allows the departing engineer’s principal to assume them. A role trust policy is not a permission grant to the person, but it is an access path. Remove the principal from the trust policy if the role was created for that person.

GCP

Remove the principal from all allow policies where it appears. Because of policy inheritance, check the organization, folder, and project levels. The Google Cloud IAM overview notes that the IAM API is eventually consistent: a write followed immediately by a read may return an older version, and changes may take time to affect access checks. Do not assume a revocation is effective the moment the API returns success. Verify with a read after a short delay, or use the policy troubleshooter.

If the engineer has a service account key, delete the key. Service account keys are long-lived credentials and are a common offboarding gap because they are often created for automation and then used interactively.

Bare metal and SSH

Remove the public key from authorized_keys on every host the engineer could reach. If you use a configuration management tool, remove the key from the source of truth and let the tool converge. If you use a bastion with short-lived certificates, revoke the certificate authority’s trust for that principal if the CA supports it; otherwise, wait for the certificate to expire and confirm the expiry window.

PostgreSQL

Revoke login on the role: ALTER ROLE username NOLOGIN;. Do not drop the role immediately if it owns objects or has granted privileges; dropping a role that owns objects requires reassigning ownership first. Check pg_shdepend for dependencies. If the role has SUPERUSER, revoke it before disabling login so the change is visible in the audit log.

If the engineer had a .pgpass file or a password in a shared vault, rotate the password for any role that was shared. A shared role is a single point of failure during offboarding; if two engineers used the same database role, disabling one person’s login does not remove the other person’s access, but it also does not remove the departing engineer’s access if the password is still known.

Phase 3: Service accounts and automation (24–36 hours)

Service accounts are where offboarding breaks the rotation. A departing engineer may have created a service account for a synthetic check, a backup job, or a deployment pipeline. If that service account is disabled without replacement, the check stops running and the rotation loses a signal.

For each service account the engineer owned or used:

  • Identify what depends on it. Check CI/CD pipelines, cron jobs, Kubernetes service accounts, and monitoring integrations.
  • Create a replacement service account with the minimum permissions needed, or transfer ownership to a shared team identity.
  • Update the dependency to use the replacement.
  • Disable the old service account and delete its keys.

In GCP, service accounts are principals. The IAM overview distinguishes human users from workloads and notes that service accounts are workload principals. That distinction is useful: a service account should not be used interactively, and an interactive user should not be the sole owner of a service account.

In AWS, the equivalent is an IAM role for EC2 or a role assumed by a pipeline. The AWS documentation describes roles for Amazon EC2 as a way to provide temporary credentials to instances without storing long-term credentials. If the departing engineer created an IAM user with an access key for a script, replace it with a role.

Phase 4: Backups and recovery paths (36–44 hours)

Backup access is the most commonly missed offboarding item because it is often shared. The question is not whether the departing engineer can restore; it is whether the remaining team can restore without them.

PostgreSQL WAL archiving

PostgreSQL’s continuous archiving documentation describes WAL archiving as a sequence of segment files copied by an archive_command or an archive_library. The archive command runs under the ownership of the PostgreSQL server user. If the archive command depends on a credential that the departing engineer controlled, the archive will fail after offboarding. Check the command for references to a specific home directory, an SSH key, or a cloud credential file.

The documentation also notes that if the archive command fails repeatedly, the pg_wal/ directory will fill with segment files, and if the file system fills, PostgreSQL will perform a PANIC shutdown. That is a production outage caused by an offboarding gap. Verify that the archive command uses a credential owned by the service, not by a person.

Test the restore path. A backup that has never been restored is a hypothesis. The Gray Haven article Write the Recovery Checklist Before You Need It describes the practice of writing the recovery procedure before an incident; use it here to confirm that the remaining team can perform a point-in-time recovery without the departing engineer’s credentials.

restic and pgBackRest

If restic repositories are encrypted with a password that the departing engineer knows, rotate the repository password. restic supports multiple keys per repository; add a new key for the team, then remove the old key. If pgBackRest uses a repository host with SSH keys, replace the key and update the configuration.

If the backup repository is in cloud storage, check the bucket or container policy for any principal that matches the departing engineer. In GCP, a bucket policy is an allow policy on the bucket resource; inherited policies from the project may also apply. In AWS, check the bucket policy and any IAM role that grants access.

Phase 5: PagerDuty and the on-call rotation (44–48 hours)

The pager identity is the last thing to remove because it is the thing that keeps the rotation working. Removing a user from PagerDuty before updating the schedule can create a gap; updating the schedule before removing the user can leave a stale notification path.

Sequence:

  1. Add a replacement responder to every schedule the departing engineer was on. If the team is small, this may mean the remaining engineers absorb the shift. Confirm that the replacement has the same escalation policy coverage.
  2. Update escalation policies to remove the departing engineer as a step. If the engineer was the only person in a step, replace the step before removing the user.
  3. Check for schedule overrides. A departing engineer may have an override that extends beyond their last day.
  4. Remove the user from PagerDuty. If the user is also the account owner or billing contact, transfer those roles first.
  5. Verify that a test incident routes to the remaining responders. PagerDuty’s integrations documentation describes email and API integrations; use a test event through the same path your production alerts use.

If the departing engineer was the only person who knew how to acknowledge a specific alert, that is a runbook gap, not a PagerDuty gap. Fix the runbook before the last day.

Break-glass access

Break-glass access is the credential you use when normal access is unavailable. It is also the credential most likely to be tied to a person. If the departing engineer held the break-glass credential, rotate it. If the break-glass credential is a shared password in a vault, change the password and update the vault entry.

The CISA Zero Trust Maturity Model page was not retrievable at the time of writing, so this article does not cite it. The operational principle is narrower: break-glass access should be auditable and time-bound. If your break-glass path is a long-lived IAM user with an access key, replace it with a role that requires MFA and issues temporary credentials. AWS documents temporary credentials as short-term and expiring; that property limits the window in which a lost break-glass credential can be used.

What to verify before closing the ticket

  • The departing engineer cannot authenticate to AWS, GCP, or any bare-metal host.
  • No service account or automation job depends on a credential the engineer controlled.
  • WAL archiving is still running and the archive command does not reference a personal credential.
  • A restore drill has been performed by a remaining team member within the last quarter.
  • The PagerDuty schedule has no gaps and a test incident routes correctly.
  • Break-glass credentials have been rotated.
  • The inventory and the actions taken are recorded in a decision record or ticket.

FAQ

Do we need to revoke temporary credentials explicitly?

No. AWS documents temporary credentials as expiring after a configured lifetime, after which AWS no longer recognizes them. The action required during offboarding is to prevent new temporary credentials from being issued, which means disabling the identity in the IdP or removing the principal from the role trust policy.

What if the departing engineer is the only person who knows the backup password?

That is a single point of failure that predates the offboarding. Rotate the password before the last day, add a new key for the team, and verify a restore. If the password is lost, the backup may be unrecoverable; treat that as an incident and document it.

How do we handle a shared database role?

Replace it with individual roles. A shared role cannot be revoked for one person without affecting others. If individual roles are not feasible, rotate the password and distribute it only to the remaining team, then plan to split the role.

What if the engineer leaves before we finish the checklist?

Prioritize human access removal and PagerDuty coverage. Service accounts and backup credentials can be rotated after the last day, but an active human credential and an uncovered pager are immediate risks. If the engineer is cooperative, ask them to transfer ownership of service accounts before their last day.

Does NIST SP 800-53 require a specific revocation timeframe?

No. SP 800-53 Rev. 5 includes an Access Control family covering account management, but the specific requirements and any timeframe are organizational decisions. The 48-hour window in this article is a team-level target, not a NIST requirement.

Sources