Incidents

An incident is a fired alert that's been routed to a channel. The Incidents page is where the on-call lives during a problem.

The list

Five filter tabs: All | Open | Investigating | Mitigated | Resolved.

Columns:

  • Title — incident summary, click to open detail.
  • Severity — one of Critical, High, Medium, Low.
  • Status — current lifecycle state.
  • Group labels — what tied alerts together. Common: service=payments, cluster=prod.
  • Acknowledged — who acked, with a clock icon if their ack expired.
  • Alerts — number of underlying alert events grouped into this incident.
  • Created — when the first alert in the group fired.

Incident severity and alert severity are different fields

They use different value sets, and confusing them is the most common mistake on this page:

FieldValuesSet by
Alert rule severitycritical, warning, infoThe alert rule, and carried as a label on each alert event
Incident severitycritical, high, medium, lowplus warning and info in practiceSet on the incident, and editable there

Why an incident can show "Warning" even though it is not in the list. When an alert raises an incident, the alert's severity label is copied onto the incident as-is, with no translation and no validation. So an incident created from a warning alert really does carry warning, and one from an info alert carries info — values the four-name list above does not contain.

In practice that means most incidents carry an alert-space severity, because most incidents come from alert rules. The four-name set applies to incidents you create or re-classify by hand.

You will therefore see six severities in the wild:

SeverityWhere it comes from
criticalBoth spaces — an alert rule, or set by hand
high, medium, lowSet on the incident by hand
warning, infoCopied from the alert rule that raised it

So filter by whichever value you actually see on the incident. If you are looking for warning-level alert noise, filter by Warning — not High or Medium.

Lifecycle states

  • Open — incident is firing. Notifications going out per the routing policy's renotify schedule.
  • Investigating — someone's looking. Notifications still go out (unless silenced).
  • Mitigated — workaround applied, root cause not fixed yet. Notifications stop. The incident reopens automatically if the underlying alert fires again.
  • Resolved — fully done. Notifications stop. Reopens only if the alert fires again within the routing policy's reopen window.

Acknowledging

Click an incident → Acknowledge in the detail page. Optionally add a comment ("looking into it", "investigating database").

Ack does three things:

  1. Moves the status from Open to Investigating (if it was Open) and records who acknowledged and when.
  2. Stops renotification until the ack timeout elapses. After that the ack expires and renotification resumes — useful if the engineer walks away from a long incident.
  3. Stamps your name + comment on the timeline.

The timeout comes from the routing policy that matched. On the seeded Default policy it is 30 minutes. If the policy cannot be read, or its value is zero, ObserveKit falls back to 30 minutes.

Reopen behavior

If an alert resolves and then fires again within the matched routing policy's reopen window, the existing incident reopens with reopened_count incremented. Beyond the window, a fresh incident is created. On the seeded Default policy the window is 5 minutes.

This avoids a flapping alert creating dozens of incidents.

Two different defaults, and which one you get

WhereAck timeoutReopen window
The seeded Default routing policy30 min5 min
The New policy form's pre-filled values60 min120 min

Both numbers are real; they just belong to different objects. Everything routed by the out-of-the-box Default policy uses 30 / 5. A policy *you* create starts at 60 / 120 because that is what the form pre-fills — and the form always submits whatever is in those boxes, so the value you see is the value you get.

Read the live numbers off each policy row in Alerts → Routing, which prints ack timeout: Nm · reopen: Nm, rather than assuming a default.

Detail page

Each incident's detail page has:

  • Header: title, severity, status, action buttons.
  • Group labels: as a label=value pill stack.
  • Routed to: the policy name that matched, with channels.
  • Ack history: who acked, when, comment. Includes ack expiries.
  • Timeline: every alert event (firing, resolved, reopened) with timestamps and values.
  • Notification log: every dispatch attempt — channel, status (success / retry / failed / suppressed), and reason if suppressed.
  • Open in Logs: deep-link into the log explorer scoped to the alert's source and time range.
  • Post-Mortem: a free-text editor stored on the incident. See Working with incidents.

Bulk actions

Checkboxes on the list let you bulk-acknowledge, bulk-mitigate, or bulk-resolve. Useful when a deploy creates a swarm of related incidents you want to clear at once.

What incidents are *not*

  • Incidents are not tickets. They're a runtime state. If you need a ticket, integrate with Jira or ServiceNow via a webhook channel that creates one on incident open.
  • Incidents are not chat threads. Use the deep-link to take the conversation to your incident-management Slack/Teams channel.

Incidents do carry a post-mortem: the detail page has an editor for it, and what you write is stored on the incident itself. Resolving an incident does not prompt you for one, so writing it stays a deliberate act — but it belongs here, next to the timeline and the notification log, rather than in a separate document nobody can find six months later.

See also

The Incidents page covers the list UI, filters/search, bulk actions, and manually opening an incident (+ New Incident).