Incident response
The life of an incident
What happens from the first alert to the all-clear, and what actually stops the paging.
Think of WarnFire as a very persistent phone tree. When something breaks, it starts calling people — and it does not stop until a real person says “I’ve got this.”
Every incident moves through three states:
1. Triggered — “something is wrong”
A monitoring system reports a problem, and WarnFire opens an incident. It looks up who is on call, takes a snapshot of the escalation plan, and starts paging.
- The first responder gets a push notification on their phone.
- If they don’t respond within about 15 seconds, WarnFire also sends them a text message.
- If nobody acknowledges before the step’s waiting time runs out, WarnFire moves to the next person in the escalation policy — and keeps going, step by step.
If the same problem is reported again while the incident is open, WarnFire recognizes it (using its dedup key — a label that identifies the problem) and adds an update to the existing incident instead of opening a duplicate. Nobody gets paged twice for one problem.
2. Acknowledged — “a person owns this”
Acknowledging is a person saying “I’ve seen it, I’m on it.” It stops the phone tree.
Only a real, signed-in person can acknowledge:
- in the mobile app (confirmed with Face ID, Touch ID, or passcode),
- in the web console (the Acknowledge button on the incident),
- or in Slack, if your workspace is connected and that person is a linked responder.
Two things that deliberately do not stop the paging:
- An “acknowledge” sent by a machine. If your monitoring system sends an acknowledge event, WarnFire writes it into the incident’s history — but keeps paging. Software saying “noted” is not the same as a person taking ownership.
- Anything from an unauthorized person. Only responders on the incident’s escalation plan can acknowledge it.
If every step runs out without anyone acknowledging, the incident is marked escalation exhausted. It stays open and highly visible — it just has no one left to page.
3. Resolved — “the problem is fixed”
Resolving closes the incident. It can come from two places:
- Your monitoring system notices the problem has cleared and sends a resolve event (Alertmanager does this automatically when you enable
send_resolved). - A person presses Resolve incident in the console.
Once resolved, the incident and its complete history stay in your records. If the same problem happens again later, a fresh incident is opened.
The history never lies
Every incident keeps an immutable timeline — a permanent, unchangeable record of what happened and when: when it triggered, who was paged on which channel, who acknowledged, what updated, and when it was resolved.
The escalation plan is frozen the moment an incident opens. Editing your schedules or policies afterward affects future incidents, never one already in flight — so the record always shows the plan that actually ran.
One quiet exception: maintenance windows
If you’ve scheduled maintenance for a service, incidents still open and record everything — but paging politely waits until the maintenance window ends. The incident shows “Paging paused for maintenance” with the resume time. See Maintenance windows.