Who Owns the Service Account? Mapping Non-Human Credentials Before the Token Expires on a Saturday

by ·Comments Off on Who Owns the Service Account? Mapping Non-Human Credentials Before the Token Expires on a Saturday

At 02:14 on a Saturday, a backup job fails. The error is not a disk failure or a network partition. It is an authentication error: the token used by the backup service account has expired, and no one on the on-call rotation knows which human owns the account, which system issued the token, or how to rotate it without breaking the restore path. The incident is not a breach. It is a mapping failure.

For lean technical teams — 2 to 15 engineers who are their own on-call rotation, running production on AWS, GCP, or bare metal with no dedicated SRE department — non-human credentials are the quiet majority of identities. They outnumber human users, they rarely appear in onboarding checklists, and they are often discovered only when they stop working. This article is about mapping them before that Saturday.

What counts as a non-human credential

A non-human credential is any identity used by a workload, job, or automation rather than a person. In practice, that includes:

  • AWS IAM roles attached to EC2 instances or ECS tasks, which supply temporary credentials that update automatically before expiry (AWS documentation, Use an IAM role to grant permissions to applications running on Amazon EC2 instances).
  • GCP service accounts, which are identified by email address and can authenticate as themselves or be impersonated by other principals (Google Cloud documentation, Service accounts overview).
  • PostgreSQL WAL archive commands that run under the ownership of the same user as the PostgreSQL server, and therefore depend on that user’s environment and credentials (PostgreSQL documentation, Continuous Archiving and Point-in-Time Recovery).
  • Restic repository passwords and access keys used by forget and prune operations, which lock the repository during pruning and require credentials that can delete data (restic documentation, Removing backup snapshots).
  • PagerDuty escalation keys, synthetic check API tokens, and monitoring webhook secrets.

The common thread is that each credential has a lifecycle: creation, use, rotation, and revocation. If no human is named as its owner, that lifecycle is unmanaged.

Why ownership mapping fails in lean teams

In a team of five engineers, the person who created a service account is often the only one who knows why it exists. When that person leaves, the account becomes an orphan. The failure mode is not dramatic. It is a slow accumulation of accounts that no one can safely delete because no one can prove they are unused.

NIST SP 800-53 Rev. 5 provides a control catalog that includes identification and authentication controls, but it does not prescribe a specific inventory format for non-human credentials. The control family structure implies that credentials should be managed, but the operational work of mapping them falls to the team. The CISA/NSA Enduring Security Framework guidance for software supply chain security notes that dependencies that are inadequately communicated or addressed may lead to vulnerabilities, and that poor supplier enterprise or development hygiene is a risk area. The same logic applies internally: an undocumented service account is an uncommunicated dependency.

The practical consequence is that token expiry becomes an incident rather than a maintenance task. AWS documentation states that temporary credentials from instance metadata update automatically before they expire, and that applications caching those credentials should refresh them every hour or at least 15 minutes before expiry. That automatic refresh works only if the application is running and the instance profile is intact. If the instance is stopped, the role is detached, or the application caches credentials beyond the refresh window, the next start can fail.

GCP service accounts present a different failure mode. Google Cloud documentation warns that service account keys are a security risk if not managed correctly, and recommends short-lived credentials or impersonation where possible. But keys still exist in many lean environments because they are easy to create and hard to inventory. A key created for a one-time migration can remain active for years if no owner is named.

A minimum viable inventory

An inventory does not need to be a database. A version-controlled YAML file or a spreadsheet with a defined schema is sufficient if it is updated as part of the change process. The fields that matter are the ones that answer the Saturday question: who do I call, what does this do, and how do I rotate it?

Recommended fields:

  • Credential identifier: the ARN, service account email, or secret name.
  • Human owner: a named engineer, not a team alias. This is the person who can authorize rotation or deletion.
  • Purpose: one sentence describing what workload uses it and why.
  • Creation date: when it was created, and by whom if known.
  • Rotation cadence: how often it should be rotated, or whether it is automatically rotated by the platform.
  • Last-used evidence: a link to a log query, a CloudTrail event, or a monitoring check that shows recent use. This is the field that prevents orphan accumulation.
  • Dependencies: what breaks if this credential is revoked. For a PostgreSQL WAL archive command, that includes the archive storage path and the restore procedure. For a restic repository, that includes the prune schedule and the lock behavior during pruning.
  • Break-glass procedure: how to rotate or revoke the credential outside normal change windows.

The last-used evidence field is the one most often omitted. Without it, deletion is a guess. With it, deletion is a decision.

Mapping the backup path specifically

Backup credentials deserve separate attention because they are both critical and often over-privileged. PostgreSQL continuous archiving requires a continuous sequence of archived WAL files that extends back at least as far as the start time of the base backup. The archive command runs under the ownership of the PostgreSQL server user, and PostgreSQL documentation advises that the archived data be protected from prying eyes, for example by archiving into a directory without group or world read access. That means the credential used by the archive command must have write access to the archive location, and the human owner of that credential must be known.

Restic adds another layer. The forget and prune commands require credentials that can delete data. The documentation notes that during a prune operation, the repository is locked and backups cannot be completed. It also advises running restic check after pruning to detect damage to internal data structures. If the credential that runs prune is owned by no one, a failed prune can leave the repository locked or the retention policy unenforced. The inventory should record which credential runs prune, when it runs, and who is responsible for verifying the result.

For a practical procedure on writing the recovery steps before an incident, see Write the Recovery Checklist Before You Need It. That article covers the checklist itself; this one covers the credential ownership that makes the checklist executable.

Offboarding and break-glass

When an engineer leaves, the offboarding checklist typically covers their human account, their email, and their laptop. It rarely covers the service accounts they created. The failure mode is not immediate. It appears weeks or months later when a token expires or a rotation is due.

The control that addresses this is a pre-offboarding review of the inventory filtered by owner. Before the last day, the departing engineer should transfer ownership of each credential to a named remaining engineer. If a credential has no remaining use case, it should be revoked as part of offboarding, not left for later.

Break-glass access for non-human credentials is a separate problem. AWS documentation describes the iam:PassRole permission, which controls which roles a user can pass to an EC2 instance. Restricting this permission is a way to prevent a user from launching an instance with more permissions than they have. In a lean team, the break-glass procedure should specify which human can assume which role, under what conditions, and how the action is logged. The inventory should record the break-glass path for each critical credential, not just the normal rotation path.

What evidence would show this works

The claim that an inventory prevents Saturday token-expiry incidents is not a research finding. It is an operational hypothesis. The evidence that would support it is specific and local:

  • A reduction in the number of authentication-failure alerts that occur outside business hours.
  • A decrease in the time to resolve credential-related incidents, measured from alert to rotation.
  • An increase in the proportion of credentials with a named human owner and a last-used evidence link.
  • A successful restore lottery — a scheduled test where a random backup is restored using only the documented procedure and the inventoried credentials.

These are not benchmarks. They are the metrics a team can collect from its own incident log and inventory. The point is to make the mapping visible before the token expires, not to prove a universal rule.

FAQ

How often should non-human credentials be rotated?
There is no single answer. AWS IAM roles for EC2 supply temporary credentials that update automatically, so rotation is handled by the platform. GCP service account keys do not rotate automatically and should be rotated on a schedule the team defines, or replaced with short-lived credentials where possible. The inventory should record the cadence and the owner responsible for enforcing it.

What is the minimum viable inventory for a team of five?
A version-controlled file with the fields listed above. The critical fields are human owner, purpose, last-used evidence, and break-glass procedure. If those four are present and current, the inventory is useful. If they are absent, it is a list of names.

How do we handle a service account that no one claims?
Treat it as a finding. Query the last-used evidence. If there is no evidence of use in a defined period, propose revocation. If there is evidence but no owner, assign an owner before the next rotation window. The goal is not to delete everything unclaimed; it is to ensure that every credential has a human who can answer for it.

Does this apply to bare-metal environments?
Yes. The credential may be an SSH key, a database password, or a restic repository password stored in a configuration file. The ownership question is the same: who rotates it, who revokes it, and who is called when it fails at 02:14 on a Saturday.

Sources