Incidents
An incident is a fired alert that's been routed to a channel. The Incidents page is where the on-call lives during a problem.
The list
Five filter tabs: All | Open | Investigating | Mitigated | Resolved.
Columns:
- Title — incident summary, click to open detail.
- Severity — one of Critical, High, Medium, Low.
- Status — current lifecycle state.
- Group labels — what tied alerts together. Common:
service=payments,cluster=prod. - Acknowledged — who acked, with a clock icon if their ack expired.
- Alerts — number of underlying alert events grouped into this incident.
- Created — when the first alert in the group fired.
Incident severity and alert severity are different fields
They use different value sets, and confusing them is the most common mistake on this page:
| Field | Values | Set by |
|---|---|---|
| Alert rule severity | critical, warning, info | The alert rule, and carried as a label on each alert event |
| Incident severity | critical, high, medium, low — plus warning and info in practice | Set on the incident, and editable there |
Why an incident can show "Warning" even though it is not in the list. When an alert raises an incident, the alert's severity label is copied onto the incident as-is, with no translation and no validation. So an incident created from a warning alert really does carry warning, and one from an info alert carries info — values the four-name list above does not contain.
In practice that means most incidents carry an alert-space severity, because most incidents come from alert rules. The four-name set applies to incidents you create or re-classify by hand.
You will therefore see six severities in the wild:
| Severity | Where it comes from |
|---|---|
critical | Both spaces — an alert rule, or set by hand |
high, medium, low | Set on the incident by hand |
warning, info | Copied from the alert rule that raised it |
So filter by whichever value you actually see on the incident. If you are looking for warning-level alert noise, filter by Warning — not High or Medium.
Lifecycle states
- Open — incident is firing. Notifications going out per the routing policy's renotify schedule.
- Investigating — someone's looking. Notifications still go out (unless silenced).
- Mitigated — workaround applied, root cause not fixed yet. Notifications stop. The incident reopens automatically if the underlying alert fires again.
- Resolved — fully done. Notifications stop. Reopens only if the alert fires again within the routing policy's reopen window.
Acknowledging
Click an incident → Acknowledge in the detail page. Optionally add a comment ("looking into it", "investigating database").
Ack does three things:
- Moves the status from Open to Investigating (if it was Open) and records who acknowledged and when.
- Stops renotification until the ack timeout elapses. After that the ack expires and renotification resumes — useful if the engineer walks away from a long incident.
- Stamps your name + comment on the timeline.
The timeout comes from the routing policy that matched. On the seeded Default policy it is 30 minutes. If the policy cannot be read, or its value is zero, ObserveKit falls back to 30 minutes.
Reopen behavior
If an alert resolves and then fires again within the matched routing policy's reopen window, the existing incident reopens with reopened_count incremented. Beyond the window, a fresh incident is created. On the seeded Default policy the window is 5 minutes.
This avoids a flapping alert creating dozens of incidents.
Two different defaults, and which one you get
| Where | Ack timeout | Reopen window |
|---|---|---|
| The seeded Default routing policy | 30 min | 5 min |
| The New policy form's pre-filled values | 60 min | 120 min |
Both numbers are real; they just belong to different objects. Everything routed by the out-of-the-box Default policy uses 30 / 5. A policy *you* create starts at 60 / 120 because that is what the form pre-fills — and the form always submits whatever is in those boxes, so the value you see is the value you get.
Read the live numbers off each policy row in Alerts → Routing, which prints ack timeout: Nm · reopen: Nm, rather than assuming a default.
Detail page
Each incident's detail page has:
- Header: title, severity, status, action buttons.
- Group labels: as a label=value pill stack.
- Routed to: the policy name that matched, with channels.
- Ack history: who acked, when, comment. Includes ack expiries.
- Timeline: every alert event (firing, resolved, reopened) with timestamps and values.
- Notification log: every dispatch attempt — channel, status (success / retry / failed / suppressed), and reason if suppressed.
- Open in Logs: deep-link into the log explorer scoped to the alert's source and time range.
- Post-Mortem: a free-text editor stored on the incident. See Working with incidents.
Bulk actions
Checkboxes on the list let you bulk-acknowledge, bulk-mitigate, or bulk-resolve. Useful when a deploy creates a swarm of related incidents you want to clear at once.
What incidents are *not*
- Incidents are not tickets. They're a runtime state. If you need a ticket, integrate with Jira or ServiceNow via a webhook channel that creates one on incident open.
- Incidents are not chat threads. Use the deep-link to take the conversation to your incident-management Slack/Teams channel.
Incidents do carry a post-mortem: the detail page has an editor for it, and what you write is stored on the incident itself. Resolving an incident does not prompt you for one, so writing it stays a deliberate act — but it belongs here, next to the timeline and the notification log, rather than in a separate document nobody can find six months later.
See also
The Incidents page covers the list UI, filters/search, bulk actions, and manually opening an incident (+ New Incident).