Setting up channels (Slack, Teams, PagerDuty, Email, Webhook)

Each notification channel needs a value from the destination system. This page covers how to get that value and which ObserveKit config field it goes in. Add channels in Settings → Notification Channels (see Notification channels).

Slack

1. In Slack, create an Incoming Webhook (Slack app → Incoming Webhooks → Add New Webhook to Workspace → pick the channel → Allow).
2. Copy the webhook URL (https://hooks.slack.com/services/…).
3. In ObserveKit: channel type Slack, set webhook_url to that URL.

Optional: timeout_seconds (default 10).

Microsoft Teams

1. In the target Teams channel, add an Incoming Webhook connector (channel … → Connectors / Workflows → Incoming Webhook), name it, and Create.
2. Copy the generated URL.
3. In ObserveKit: channel type Teams, set webhook_url. (Notifications render as Adaptive Cards.)

PagerDuty

1. In PagerDuty, open the Service you want to page → Integrations → add an Events API v2 integration.
2. Copy the Integration Key (32 characters, begins R0…).
3. In ObserveKit: channel type PagerDuty, paste it into Integration Key.
4. Save the channel, then click Send test on its row. The test action lives on a saved channel, not in the create dialog — it needs the stored channel to send as, so there is no way to test before saving. A wrong key fails at PagerDuty's ingest, and you want to discover that now rather than during an incident. A successful send returns PagerDuty's 202 Accepted.

Send test opens a real incident on your service, then closes it. PagerDuty has no "ping" — the only way to prove the key works is to trigger something. So ObserveKit sends a trigger and immediately follows it with a resolve on the same dedup key. Expect a brief page that self-clears; on most services that is a notification rather than a call, but it is a real incident and it will appear in your PagerDuty timeline.

If the resolve fails, the response says so explicitly and names the dedup key (test-dedup) — that means a test incident is still open and you need to resolve it by hand in PagerDuty.

That is the whole setup. ObserveKit talks to https://events.pagerduty.com/v2/enqueue directly; there is no URL to configure and no payload mapping to write on the PagerDuty side.

What PagerDuty receives

Every event carries the incident's `dedup_key`, so PagerDuty groups the whole lifecycle onto one page instead of opening a new one per event — and a resolve closes it automatically rather than leaving a page for someone to clear by hand.

> Known limitation — a page opened by an ESCALATION step is not auto-resolved.

>

> If an escalation policy pages a channel that

> is not also attached to the incident's routing policy, that channel receives

> the trigger and never the matching resolve. The PagerDuty incident stays

> open until someone clears it by hand, even though ObserveKit shows the incident

> resolved.

>

> This affects the default setup, since a new escalation step targets a channel,

> and a user target resolves to that person's own preference channels — which

> are by definition not the policy's.

>

> **Until it is fixed, attach any PagerDuty channel you escalate to as a channel

> on the routing policy as well.** Then the resolve reaches it by the normal path.

> Tracked as todo/0028 in the repository.

Severity maps onto PagerDuty's scale:

This matters if your PagerDuty service is configured to page only above a threshold — a low ObserveKit incident arrives as info and, on most services, will not wake anyone. That is usually what you want, but check it against your service's rules rather than assuming.

The Integration Key is a credential. It grants the ability to open incidents on that service — anyone holding it can page your on-call.

Be precise about how well it is protected, because "masked" is not "unreadable":

So it is admin-readable, not write-only. Every route involved is admin-only, so this is not an escalation path — but treat the key as recoverable by anyone with admin access, and rotate it in PagerDuty if that assumption stops holding. Narrowing this is tracked as todo/0027 in the repository.

When to use a Webhook instead

Point a Webhook channel at PagerDuty's endpoint only if you need to reshape the event on the way — a different summary, extra custom details, or routing through your own translator. You then own the payload and the resolve behaviour, both of which the native type handles for you.

If your receiver cannot be adapted to ObserveKit's payload shape, put a small translator in front of it — the webhook body is stable and documented, and translating it is a few lines.

Email (SMTP)

Channel type Email. Config fields:

Common providers:

• Gmail / Google Workspace: smtp.gmail.com : 587, use_tls: true, username = the address, password = an App Password (not your login password).
• Microsoft 365: smtp.office365.com : 587, use_tls: true.
• SendGrid: smtp.sendgrid.net : 587, username = apikey (literally), password = your SendGrid API key.

Webhook (custom receiver)

Channel type Webhook. Fields: webhook_url (required), secret (optional, enables signing), header_<Name> (optional, forwards a custom header), timeout_seconds (default 10).

Payload

ObserveKit POSTs JSON:

{
  "event_type": "IncidentCreated",
  "incident_id": "...",
  "title": "High error rate on payments",
  "severity": "critical",
  "status": "open",
  "dedup_key": "...",
  "auto_action": false,
  "timestamp": "2026-07-09T12:00:00Z",
  "actor_id": "...",
  "summary": "...",
  "labels": { "service": "payments", "cluster": "prod" },
  "deep_link": "https://observekit.expeed.com/incidents/<id>"
}

(auto_action is a raw boolean; timestamp is RFC3339; labels is an object.)

Verifying the signature (HMAC)

If you set a secret, ObserveKit signs each request:

• Header: `X-ObserveKit-Signature`
• Value: `sha256=<hex>` where <hex> is HMAC-SHA256 of the raw request body using your secret.

Verify by recomputing over the exact bytes you received (don't re-serialize the JSON) and comparing:

expected = "sha256=" + hex(hmac_sha256(secret, raw_body))
constant_time_equal(expected, X-ObserveKit-Signature)

No secret → no signature header is sent.

After setup

Use Send test on the channel to confirm delivery before wiring it into a routing policy. Not getting notifications? See Troubleshooting.