Channels and notifications
A channel is somewhere ObserveKit can send a notification. Email inbox, Slack webhook, PagerDuty integration. Channels are managed in Settings → Notification Channels, not on the Alerts page — they're infrastructure config, shared across every routing policy.
Five channel types
| Type | Best for | Required config |
|---|---|---|
| Audit trail, weekly digests, low-frequency alerts | SMTP host, port, username, password, from-address, default to-addresses | |
| Slack | Team channel notifications | Incoming webhook URL |
| Teams | Team channel notifications (Microsoft) | Incoming webhook URL |
| Webhook | Custom integrations, ITSM tools | URL, optional HMAC secret, optional headers |
| PagerDuty | High-severity paging | Events API v2 integration key |
Adding a channel
- Go to Settings → Notification Channels.
- Click + Add channel.
- Pick the type. The form switches to show the right fields for that type.
- Fill in the type-specific config.
- Send test — fires a synthetic notification through the channel so you can verify it lands. Save only after the test succeeds.
- Save.
The channel now appears in the routing policy form's Send notifications to picker.
Editing & deleting
Settings has pencil and trash icons on every channel row. Edits take effect immediately on the next dispatch. Deleting a channel removes it from any routing policies that referenced it; those policies still work but with one fewer recipient.
Per-channel rate limit
Notifications are rate-limited to 10 per minute per channel by default. This is a server-wide setting (alerting.notifier.rate_limit.per_channel_per_min), not a per-channel-row field. Excess notifications are queued and labelled rate_limited in the audit log.
This protects against runaway alert storms. If a thousand pods all fail at once, your Slack channel gets ten messages per minute — not a thousand.
Retry policy
Failed notifications retry 3 times with exponential backoff (1s, 2s, 4s). After 3 failures the notification is marked retry_exhausted and skipped — but the alert itself stays firing. Other channels still get notified normally.
Common causes of retry_exhausted:
- Slack/Teams webhook URL revoked.
- SMTP credentials rotated.
- PagerDuty integration disabled.
- Network outage between ObserveKit and the channel target.
Check Settings → Notification Channels for the channel's recent send status, and resend a test to confirm it's working.
What a notification looks like
The body of a notification is generated by ObserveKit using a state-aware template. It includes:
- Incident title and severity.
- Group labels (e.g.,
service=payments,cluster=prod). - Summary and description from the rule.
- Direct deep-link to the incident in ObserveKit.
- For email: a runbook link if the rule provides one.
- For Slack/Teams: a button block linking back to ObserveKit.
- For PagerDuty: an Events API v2 payload with
dedup_keyso PagerDuty groups follow-ups correctly.
Customising templates is on the roadmap. For now, every channel of a given type uses the same default template.