A service whose job begins
when you stop answering.
Which makes the scheduled job the product.
People store messages and the contacts they are meant for. The service sends a heartbeat mail. If someone stops answering it, their messages go to the people they chose. Alongside that runs a public donor-status card, looked up by a short key.
None of this is technically exotic. What makes it a case file is the operating standard the promise imposes: a background job that quietly stops is not degraded service. It is the whole failure, and nobody would notice from the outside.
- Status
- Live since 2023
- Kind
- Consumer service
- Reach
- 4 languages
- Stack
- PHP · MariaDB · REST
- Critical path
- Scheduled dispatch
- Deploy
- Gated, smoke-tested
- probe account/login 200Answering. Checked on every run, because a login that fails silently locks people out of their own switch.
- probe account/register 200Answering.
- probe account/status 200Answering.
- probe scheduled dispatch staleOverdue. This is the heartbeat — nothing user-facing looks wrong while it is down.
- recovery healthcheck→the fleet inbox resolvedBack within the minute, and the recovery is posted as loudly as the failure was.
- release deploy gate blockedA release only proceeds when the recorded commit matches and the smoke test comes back clean.
A landing page returning 200 proves nothing about this product. The probe asks the three endpoints a user needs and, separately, whether the scheduled job is still fresh.
Brief What the promise costs § 01
Most outages annoy people. This one would go unnoticed.
A shop that goes down loses orders and someone complains within minutes. A dead-man's switch that goes down looks exactly like a dead-man's switch that is working. Nobody is waiting for output. The failure only becomes visible on the one day it matters, to the one person who cannot be told.
That asymmetry is the whole engineering brief. It means monitoring is not an operational nicety bolted on after launch — it is a functional requirement of the product, and it has to watch the part users never see.
The service has run since 2023, in four languages, with legal guidance pages for several jurisdictions. Everything below follows from taking that one asymmetry seriously.
Ledger What is watched § 02
Decisions What we chose, and what it cost § 03
Three calls that follow from one asymmetry.
Watch the heartbeat, not the homepage.
An uptime check on a landing page would have been green through every incident this product has had. So the probe asks the three account endpoints a real user depends on, and separately checks whether the scheduled dispatch is still running on time.
Both the failure and the recovery are posted into the same inbox the rest of the fleet reads. Recoveries are announced as loudly as failures, because an alert nobody closes trains everyone to ignore the next one.
Cost: noise when something flaps, and a probe that itself has to be kept honest.
Bought: a minute of staleness becomes a message someone saw, instead of a promise quietly broken.
A release you cannot prove is not a release.
Deployment runs behind a one-time key that rotates, and against a recorded release commit that has to match what is being shipped. Afterwards a smoke test hits login, register and status; any server error stops the release rather than completing it.
It is more ceremony than a service this size would normally justify. It is justified here because the cost of a bad deploy is not a bad afternoon.
Cost: more steps between a fix and production.
Bought: no deployment has quietly taken the product down.
We removed a privacy feature rather than half-deliver it.
The interface offered a stricter privacy mode. It could not be honoured all the way through to dispatch without weakening the guarantee it implied. We took the option out of the interface and made the dispatch path ignore the field outright.
A privacy promise that mostly holds is worse than no promise, because the person relying on it cannot tell the difference until it matters.
Cost: a visible feature disappeared, and that is never a comfortable release note.
Bought: nothing in the interface claims a guarantee the system does not actually keep.
Incident What broke, and what it changed § 04
The failure mode was exactly the invisible one.
Availability
The scheduled job ran out of memory.
The dispatch job exhausted its memory and stopped. The website stayed up. Every page a visitor could reach answered normally. The only symptom was that the one job carrying the product's actual promise was no longer running on time.
What we had assumed: that a job which had run for years without attention would keep doing so as data grew underneath it. Workloads that never fail are simply workloads that have not yet met a large enough input.
Fix → the memory issue itself, and — more importantly — a freshness check so the absence of work became an alarm. Monitoring that only watches for errors cannot see a job that never started.
Transfer What this means where you are § 05
If your product has a silent critical path, it needs a different kind of monitoring.
Alarm on absence, not just errors
Error rates cannot see work that never started. Anything whose value is delivered by a scheduled job needs a freshness check, or your dashboard will stay green through the only outage that counts.
Announce recoveries as loudly as failures
An alert stream where nothing ever closes is an alert stream people learn to scroll past. Pairing every failure with its resolution is what keeps the channel worth reading.
Remove promises you cannot keep
Half-implemented guarantees are a liability disguised as a feature. Taking one out is a worse release note and a much better product.