Notification channels

The Notification Channels section of Settings is where channels are added, edited, tested, and deleted. Channels are consumed by routing policies (Alerts → Routing) and escalation policies (Alerts → Escalation) to deliver alert notifications.

The list

Four columns:

Adding a channel

Click + Add Channel top-right. The modal asks for the type first; the form below switches to the fields appropriate to that type.

Per-type fields

Email:

• SMTP host
• SMTP port
• Username
• Password (masked)
• From address
• Default to-addresses (comma-separated)
• Use TLS (checkbox)

Slack:

• Incoming webhook URL

Teams:

• Incoming webhook URL

PagerDuty:

• Integration Key (masked) — from an Events API v2 integration on the PagerDuty service you want to page

Webhook:

• Endpoint URL
• HMAC shared secret (masked, optional — adds an X-ObserveKit-Signature header, value sha256=<hex digest>)
• Additional headers (key/value pairs, optional)

The Integration Key is the entire configuration of a PagerDuty channel — there is no URL to set, because ObserveKit posts to PagerDuty's Events API v2 endpoint itself. See Setting up channels for where to find the key and how incidents map onto PagerDuty's severities.

Send test

Every channel has a Send test action (paper-plane icon on the row). Click it to fire a synthetic notification through the channel and verify it lands. Use this any time you change channel config, or as a sanity check during alert flow setup.

The test renders a sample alert payload titled "Test notification from ObserveKit", so it is obvious to whoever receives it that nothing is wrong.

Editing or deleting

• Pencil opens the same form populated with current values.
• Trash removes the channel. Routing policies referencing the channel keep working but with one fewer recipient.

Per-channel rate limit

Each channel has a sliding-window rate limit, default 10 notifications per minute. Excess notifications are queued and labelled rate_limited in the audit log. Protects against runaway alert storms — a thousand simultaneous pod failures still send only ten messages per minute.

The rate limit is server-wide configuration (alerting.notifier.rate_limit.per_channel_per_min in the YAML), not per-channel-row.

Retry policy

Failed notification dispatches retry 3 times with exponential backoff (1s, 2s, 4s). After 3 failures the dispatch is marked retry_exhausted and skipped — but the alert itself stays firing. Other channels in the same routing policy still get notified normally.

Common causes of retry_exhausted:

• Slack/Teams webhook URL revoked.
• SMTP credentials rotated.
• A webhook receiver returning a non-2xx status.
• A PagerDuty Integration Key that was rotated, or belongs to a deleted service — PagerDuty rejects the event and every retry fails the same way.
• Network outage between ObserveKit and the channel target.

Check the channel's recent send status (the audit log surfaces this) and resend a test to confirm it's working.

Templates

Notification body content is generated by ObserveKit using state-aware templates. Each channel type has a default template:

• Email — multi-part text + HTML.
• Slack / Teams — block-kit JSON with the alert summary, severity color stripe, group labels as fields, and a button that deep-links back to the incident in ObserveKit.
• PagerDuty — a PagerDuty Events API v2 event: trigger / acknowledge / resolve keyed on the incident's dedup key, with the ObserveKit severity mapped onto PagerDuty's scale. See Setting up channels.
• Webhook — JSON envelope with the alert event + incident metadata, including the incident's dedup key. Optional HMAC signing for receiver-side verification.

Per-tenant template customization is on the roadmap.