Notification channels

The Notification Channels section of Settings is where channels are added, edited, tested, and deleted. Channels are consumed by routing policies (Alerts → Routing) and escalation policies (Alerts → Escalation) to deliver alert notifications.

The list

Four columns:

Adding a channel

Click + Add Channel top-right. The modal asks for the type first; the form below switches to the fields appropriate to that type.

Per-type fields

Email:

• SMTP host
• SMTP port
• Username
• Password (masked)
• From address
• Default to-addresses (comma-separated)
• Use TLS (checkbox)

Slack:

• Incoming webhook URL

Teams:

• Incoming webhook URL

Webhook:

• Endpoint URL
• HMAC shared secret (masked, optional — adds an X-Signature header)
• Additional headers (key/value pairs, optional)

PagerDuty:

• Events API v2 integration key (masked)

Send test

Every channel has a Send test action (paper-plane icon on the row). Click it to fire a synthetic notification through the channel and verify it lands. Use this any time you change channel config, or as a sanity check during alert flow setup.

The test renders a sample alert payload with a clearly synthetic title (e.g., "ObserveKit test notification — please ignore") so it's obvious to the receiver.

Editing or deleting

• Pencil opens the same form populated with current values.
• Trash removes the channel. Routing policies referencing the channel keep working but with one fewer recipient.

Per-channel rate limit

Each channel has a sliding-window rate limit, default 10 notifications per minute. Excess notifications are queued and labelled rate_limited in the audit log. Protects against runaway alert storms — a thousand simultaneous pod failures still send only ten messages per minute.

The rate limit is server-wide configuration (alerting.notifier.rate_limit.per_channel_per_min in the YAML), not per-channel-row.

Retry policy

Failed notification dispatches retry 3 times with exponential backoff (1s, 2s, 4s). After 3 failures the dispatch is marked retry_exhausted and skipped — but the alert itself stays firing. Other channels in the same routing policy still get notified normally.

Common causes of retry_exhausted:

• Slack/Teams webhook URL revoked.
• SMTP credentials rotated.
• PagerDuty integration disabled.
• Network outage between ObserveKit and the channel target.

Check the channel's recent send status (the audit log surfaces this) and resend a test to confirm it's working.

Templates

Notification body content is generated by ObserveKit using state-aware templates. Each channel type has a default template:

• Email — multi-part text + HTML.
• Slack / Teams — block-kit JSON with the alert summary, severity color stripe, group labels as fields, and a button that deep-links back to the incident in ObserveKit.
• Webhook — JSON envelope with the alert event + incident metadata. Customizable HMAC signing for receiver-side verification.
• PagerDuty — Events API v2 payload with dedup_key set to the incident's dedup key so PagerDuty groups follow-ups correctly.

Per-tenant template customization is on the roadmap.