Outages & post-mortems
A server that stops serving for three minutes and comes back on its own often leaves no trace anyone looks at. Demerzel finds it in what the server already reported, writes the post-mortem, puts it on the server page and sends it to your channels — without you asking.
What it is built from
Two things every agent already sends, nothing new on the host:
- When Demerzel received each agent report. Between two of them, Demerzel heard nothing from the server. A gap counts when three reports in a row are missing, and never under three minutes: an agent restarting for an update is back in seconds.
- Requests per minute from the server’s own web logs, when the agent may read them (
grant-web-logs). A minute is dark when it served nothing at all: one request answered proves the server was up. Only a server that usually serves at least 10 requests a minute (the median of the hour before) is judged this way — below that, an empty minute is just a quiet night. Two dark minutes make an outage when the agent’s silence confirms them, three on their own.
What kind of outage
| Kind | What the evidence shows |
|---|---|
| Stopped answering | No visitor got in and the agent stalled, while the machine kept running (same boot). The agent checks your certificates through the local web server, so a web server whose workers are all held stalls it too. Most often an address banned mid-flight: a ban stops its new packets, not the connections it already holds, and the web server waits on each until its timeout (300 s on Apache). A network cut looks the same from the host; the report weighs both against the bans just before. |
| Sites down | No visitor got in, but the agent reported every minute: the machine and its network were fine, the web server (or what sits in front of it: load balancer, DNS, CDN) was not. |
| Restart | The server booted during the outage. |
| Not heard from | The agent could not report for at least five minutes on a server whose web logs are not read — so nobody can say whether visitors noticed. |
An agent that stalls while the web logs show visitors being served is the agent’s problem, not an outage, and is not reported. A silence shared by most of your servers at the same moment is Demerzel’s own, not theirs, and is not reported either.
The post-mortem
Each outage gets a report, written from its evidence alone — every sentence is a fact with its timestamp:
- Impact — how long visitors were cut off, about how many requests went unserved, how long the agent was not heard from, and whether the machine restarted;
- Requests per minute around it, with the dark minutes and the agent’s silence drawn behind;
- What happened, and a timeline: traffic falling, the last report before the cut, the first request back, and what the warden or fail2ban banned just before;
- What the evidence cannot tell — for a server that stopped answering, whether its workers were held or its network was cut;
- Next steps — for a server that stopped answering: how to spot the address holding the connections and free them, how to make bans close what they leave open, and, failing that, the provider’s network history to check for the exact window.
Find it on the server’s Overview tab, under Outages — last 30 days.
Who is told
Each outage raises one Action, warning — it is over, nobody needs to run — for a day after it ends. It is the one warning every channel subscribed to Availability receives whatever its threshold, and the email carries the whole post-mortem, with a link to the page. See Notifications.
An outage is found once it is over: a gap is only a gap once the next report arrives, and a minute is only known empty once a later one was served.