Channels and notifications
A channel is somewhere ObserveKit can send a notification. Email inbox, Slack webhook, HTTP endpoint. Channels are managed in Settings → Notification Channels, not on the Alerts page — they're infrastructure config, shared across every routing policy.
Five channel types
| Type | Best for | Required config |
|---|---|---|
| Audit trail, weekly digests, low-frequency alerts | SMTP host, port, username, password, from-address, default to-addresses | |
| Slack | Team channel notifications | Incoming webhook URL |
| Teams | Team channel notifications (Microsoft) | Incoming webhook URL |
| PagerDuty | Waking someone up — on-call rotations, escalation | Integration Key from an Events API v2 integration |
| Webhook | Custom integrations, ITSM tools, anything else | URL, optional HMAC secret, optional headers |
Choosing between PagerDuty and a webhook
Use the PagerDuty type. It speaks the Events API v2 directly and sends trigger and resolve actions keyed on the incident's dedup key, so a resolved ObserveKit incident closes the PagerDuty page rather than leaving it open for someone to clear by hand.
A Webhook aimed at PagerDuty's endpoint also works, but you own the payload mapping and the resolve behaviour. Reach for it only if you need to transform the event on the way — otherwise the native type is less to maintain.
Adding a channel
- Go to Settings → Notification Channels.
- Click + Add channel.
- Pick the type. The form switches to show the right fields for that type.
- Fill in the type-specific config.
- Send test — fires a synthetic notification through the channel so you can verify it lands. Save only after the test succeeds.
- Save.
The channel now appears in the routing policy form's Send notifications to picker.
Editing & deleting
Settings has pencil and trash icons on every channel row. Edits take effect immediately on the next dispatch. Deleting a channel removes it from any routing policies that referenced it; those policies still work but with one fewer recipient.
Per-channel rate limit
Notifications are rate-limited to 10 per minute per channel by default. This is a server-wide setting (alerting.notifier.rate_limit.per_channel_per_min), not a per-channel-row field. Excess notifications are queued and labelled rate_limited in the audit log.
This protects against runaway alert storms. If a thousand pods all fail at once, your Slack channel gets ten messages per minute — not a thousand.
Retry policy
Failed notifications retry 3 times with exponential backoff (1s, 2s, 4s). After 3 failures the notification is marked retry_exhausted and skipped — but the alert itself stays firing. Other channels still get notified normally.
Common causes of retry_exhausted:
- Slack/Teams webhook URL revoked.
- SMTP credentials rotated.
- A webhook receiver returning a non-2xx status (an expired integration key at the far end shows up this way).
- Network outage between ObserveKit and the channel target.
Check Settings → Notification Channels for the channel's recent send status, and resend a test to confirm it's working.
What a notification looks like
The body of a notification is generated by ObserveKit using a state-aware template. It includes:
- Incident title and severity.
- Group labels (e.g.,
service=payments,cluster=prod). - Summary and description from the rule.
- Direct deep-link to the incident in ObserveKit.
- For email: a runbook link if the rule provides one.
- For Slack/Teams: a button block linking back to ObserveKit.
- For webhook: a JSON envelope carrying the alert event and incident metadata, including the incident's dedup key — which is what a receiver such as PagerDuty needs in order to group follow-ups rather than open a new page each time.
Customising templates is on the roadmap. For now, every channel of a given type uses the same default template.