The failure you need to catch is the run that never happened

Most backup monitoring watches for errors. That is the wrong shape for a cron job. The failure modes that hurt a lean team are quieter: the crontab was overwritten by a config-management run, the cron daemon is not running in the rebuilt container, the host was powered off at 02:30, or the job started and hung on a stale NFS mount. None of those reliably produce an error you will see. They produce silence.

A dead man’s switch inverts the reporting direction. The job reports success when it finishes; the absence of that report is the alert. Healthchecks.io describes the mechanism plainly: it listens for HTTP pings from cron jobs and scheduled tasks, stays silent while pings arrive on time, and raises an alert as soon as a ping does not arrive on time. The same documentation calls this a dead man’s switch for processes that need to run on a regular, known schedule, and lists filesystem backups and database backups among the jobs it is built to monitor (Healthchecks.io documentation).

The argument here is narrower than “add monitoring.” For a team of two to fifteen engineers who are their own on-call rotation, the cheapest useful switch is not a new daemon, dashboard, or service to operate. It is one success-only HTTP call appended to the backup job you already run, plus a grace window sized to that job’s real behavior. The rest of this article is about wiring that call correctly, sizing the window so it does not page on normal runs, and being honest about what the switch does not prove.

Why cron’s built-in mail is not the switch

Cron does have a notification path. The cron manual states that job output is mailed to the owner of the crontab, or to the user named in MAILTO, and that output can also be sent to syslog with the -s option (cron(8)). That is useful for diagnosing a job that ran and printed an error. It is not a dead man’s switch.

Healthchecks.io’s cron guide lists the practical drawbacks: MAILTO requires a configured MTA on the host, it does not notify you when the whole machine is powered off or has lost network connectivity, and routine stdout produces mail on every run, which trains people to ignore it (How to Monitor Cron Jobs). Those are drawbacks of the channel, not proof that every server lacks an MTA. The point is structural: mail from the host cannot report the host being gone.

A success ping sent to an external endpoint covers the cases mail cannot. If the host is off, no ping arrives. If cron is stopped, no ping arrives. If the crontab was wiped, no ping arrives. If the job hangs, no success ping arrives. All four look identical from outside, which is exactly what you want from a switch.

The minimal wiring: one gated call

Healthchecks.io’s guide shows the pattern as a single crontab line, with the ping chained using && so it runs only when the backup exits zero:

# run backup.sh, then send a success signal
8 6 * * * /home/me/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/your-uuid-here

The guide explains the curl flags: retry failed requests, cap the maximum execution time, and silence output unless there is an error. It also notes that the ping can instead be the final line inside the backup script, which keeps the crontab short (Healthchecks.io cron guide).

For anything with more than one step, put the ping at the end of the script under set -euo pipefail. That way any failing step aborts before the ping is reached, and the ping asserts on the whole pipeline rather than the exit code of a wrapper. A backup that runs, writes nothing, and exits zero is still a failed backup; put whatever verification the job can afford before the ping, and the switch covers that too.

Two operational details matter more than they look. First, cron runs with a minimal PATH, so use the absolute path to curl. Second, treat the ping URL as a secret. Healthchecks.io’s documentation is explicit: check UUIDs and project ping keys should be treated as secrets, because anyone who has them can send telemetry signals to your checks and disturb your monitoring (Healthchecks.io documentation). Keep the URL out of public repositories and pass it through an environment variable or a secrets store.

Sizing the grace window from observed behavior

A heartbeat check has a schedule and a grace time. Healthchecks.io defines grace time as the additional wait before alerting when a check is late, intended to absorb minor expected deviations in job execution (Healthchecks.io documentation). Its cron guide adds the practical rule: set grace time above the expected duration of the job (Healthchecks.io cron guide).

For a nightly backup, that means the window should be built from three numbers you can observe: how long the job usually takes, how long the slowest recent run took, and how much schedule drift you tolerate. A round number like “one hour” is a guess. A window derived from the p95 duration of the last thirty runs plus a margin is a decision you can defend in a postmortem.

The tradeoff is symmetric. Too tight a window pages on a normal slow night and erodes trust in the alert. Too loose a window means a skipped run is not noticed until most of the next day has passed. For a daily backup, a window that fires within a few hours of the expected completion is usually the right compromise; for an hourly job, the window has to be much shorter or the switch is decorative.

If the team is migrating to systemd timers, the same sizing applies, with two extra sources of drift to account for. The systemd.timer documentation states that AccuracySec defaults to one minute, so timers do not necessarily expire at the precise configured time, and that RandomizedDelaySec adds a random delay between zero and the configured value to spread load (systemd.timer(5)). If you use either setting, the grace window must absorb it.

Notification channels and redundancy

A switch that alerts into a channel nobody watches is not a control. Healthchecks.io supports multiple integrations, including email, webhooks, SMS, Slack, and PagerDuty, and allows more than one per check (Healthchecks.io documentation). Its cron guide recommends setting up multiple notification methods for redundancy and using different channels depending on job priority (Healthchecks.io cron guide).

For a lean team, the useful configuration is usually two channels with different failure modes: one that reaches the on-call rotation (PagerDuty or an equivalent escalation path) and one that leaves a durable record in a team channel. Email alone is a single point of failure; a chat message alone is easy to miss overnight. The switch should page a human who can act, not a channel that is read the next morning.

The runbook attached to that page should name the first three checks before anyone touches the backup repository: is the host reachable, is the scheduler running, and does the job log show a start for the expected run. Those three questions separate “the backup failed” from “the backup never started,” and they are the difference between a five-minute fix and an hour of guessing.

What the switch does not prove

A success ping proves the job reached the end of its script and exited zero. It does not prove the backup is restorable. That gap is not hypothetical: a backup can upload zero bytes, write an incomplete snapshot, or archive WAL files that cannot be replayed, and still exit zero and ping.

PostgreSQL’s documentation is direct about the WAL side. Recovery using continuous archiving requires a continuous sequence of archived WAL files extending back at least to the start time of the backup, and the documentation advises setting up and testing the archiving procedure before taking the first base backup (PostgreSQL: Continuous Archiving and PITR). It also states that the archive command must return zero exit status if and only if it succeeds, and that a nonzero status causes PostgreSQL to retry periodically. A heartbeat on the backup job does not replace monitoring the archiver itself; the same documentation advises monitoring the archiving process to ensure it is working as intended.

The practical consequence is that the ping belongs after whatever verification the job can afford, and a periodic restore drill remains necessary. The switch tells you the job ran. The drill tells you the backup is worth having. Those are different controls, and a lean team needs both. For a starting point on the drill side, see Write the Recovery Checklist Before You Need It.

Optional hardening: start and fail signals

The success-only ping is the minimum. Healthchecks.io’s Pinging API also supports start, failure, and log signals, and supports reporting a script’s exit status where zero is success and any other value is failure (Pinging API). Sending a start signal is optional, but it enables measuring job execution time and detecting a job that runs longer than its configured grace time.

That last capability is worth the extra call for a backup that can hang. A start ping arms overrun detection: if the matching success does not arrive inside the grace window, the check goes down while the job is still wedged, rather than waiting for the next scheduled run to reveal the problem. The cost is one more HTTP call and a slightly more complex script. For a nightly job that occasionally stalls on a network mount, it is usually worth it.

If the team is on systemd timers, the same pattern moves cleanly. A oneshot service can use ExecStartPost for the success ping, which runs only after ExecStart has completed successfully, and an OnFailure unit for the fast-failure signal. The heartbeat still catches the timer that never fired, which is the case OnFailure cannot see. The systemd.timer documentation also notes that Persistent=true stores the last trigger time and runs a missed activation at the next opportunity, which is useful for backups but does not remove the need for the switch: a host that never comes back still produces no ping (systemd.timer(5)).

Test the switch itself

An untested switch is a belief, not a control. The failure path has to be exercised once, deliberately, before you rely on it. The procedure is short: make the backup script exit nonzero on a test run, confirm that no success ping is sent, and confirm that the alert arrives on the expected channel within the expected window. Then restore the script and confirm the next successful run clears the alert.

Two details are easy to get wrong in that test. First, a timezone mismatch between the host and the check configuration can make pings arrive an hour early or late; Healthchecks.io’s guide names this as the likely cause when pings are consistently offset (Healthchecks.io cron guide). Second, the alert window is the schedule plus the grace time, so a test that waits only for the schedule will conclude too early that the switch is broken.

After the test, write down three things in the runbook: who receives the alert, what the first three checks are, and how to pause the check during a known maintenance window. Healthchecks.io supports a paused state for exactly this purpose, so a planned outage does not generate a false page (Healthchecks.io documentation).

Frequently asked questions

Do I need a separate monitoring service for this?

No. The pattern is one HTTP call from the job to an endpoint that expects it on a schedule. That endpoint can be a hosted heartbeat service or a self-hosted equivalent. The decision is about who operates the endpoint, not about whether the pattern works. For a lean team, the tradeoff is usually between a small external dependency and the operational cost of running another service yourself.

Should the ping fire at the start or the end of the job?

The end, and only on success. A start ping proves the scheduler fired, which is the easier half of the problem. A success ping proves the work completed. If you want overrun detection as well, send both: a start signal to arm the timer and a success signal to clear it.

What if the monitoring endpoint itself is down?

That is a real dependency, and it should be named in the runbook rather than assumed away. The practical mitigations are to use a service with its own redundancy and to keep a second, independent signal for the highest-priority job, such as a daily summary that a human reads. The switch reduces the chance of silent failure; it does not eliminate every dependency.

Does this replace restore testing?

No. The switch proves the job ran and exited zero. Restore testing proves the backup is usable. A backup that exits zero and pings can still be unrestorable, so the ping should sit after any verification step the job can afford, and a periodic restore drill remains necessary.

How do I keep the ping URL out of logs and repositories?

Treat it as a secret. Pass it through an environment variable or a secrets store, keep it out of public repositories, and avoid printing it in job output. Healthchecks.io’s documentation is explicit that public ping URLs let anyone send telemetry signals to your checks and disturb your monitoring (Healthchecks.io documentation).