Troubleshooting

The most common things that go wrong, and how to fix them.

My alert isn't firing

Check, in order:

1. The rule is enabled. Rules tab → confirm the toggle is on.
2. The condition is actually true. Use the Preview button on the rule's edit form to replay against the last 24h. If preview shows zero matches, the threshold is wrong, the scope is too narrow, or the metric isn't reporting.
3. The metric is being ingested. Go to the source's logs/metrics view and confirm fresh data is flowing in.
4. The sustain window isn't longer than the spike. A 30-minute sustain on a 5-minute spike won't fire. Lower the sustain or set it to 0 for instant rules.

My alert fires but nobody is notified

1. A routing policy matches. Routing tab → click Test routing → enter the alert's labels → confirm at least one policy matches.
2. Channels are attached. Open the matching policy → confirm the Send notifications to pills are populated.
3. A silence isn't suppressing it. Silences tab → Active → check whether anything matches the alert.
4. The channel actually delivers. Settings → Notification Channels → click Send test on the channel. If the test fails, the underlying webhook/SMTP/PagerDuty integration is broken. Fix the channel config before doing anything else.
5. The notification didn't get rate-limited. Each channel has a 10-per-minute limit. The audit log shows rate_limited entries. Spread the notifications across more channels or wait.

My alert fires repeatedly for the same thing

The routing policy's Combine new alerts within window may be too short, or your Group by labels may be too specific. Increase the combine window (try 5m → 15m) or add more labels to Group by (e.g., add cluster).

Alternatively the rule's Sustained for window may be too short — a flapping metric will fire/resolve repeatedly. Increase sustain.

I'm getting paged at 3 AM for non-critical things

Check the rule's Severity. A flood of Critical for things that aren't critical is the usual cause.

Then check routing: is the Critical policy attached to PagerDuty? Should it be? If most "critical" alerts shouldn't page, the severity threshold is wrong. Demote them to Warning.

An incident reopened immediately after I resolved it

The underlying alert is still firing. Either:

• The condition is still true — the actual problem isn't fixed.
• The reopen window is wider than the alert's flap interval — the alert toggled off briefly when you resolved, then back on. The reopen window catches it.

Use Mitigated instead of Resolved for problems with workarounds in place but root cause unfixed. Mitigated incidents reopen on next fire just like Resolved, but the lifecycle status reflects reality.

I can't find a channel I added

Settings → Notification Channels — was the channel saved? Click + Add channel if it's missing. After saving, go back to the routing policy and the new channel should appear in the Send notifications to picker.

If the channel exists in Settings but not in the picker, reload the page — the form caches the channel list at open time.

A deleted rule still shows up in incidents

Incidents persist even after their rule is deleted. The Incidents page lists historical incidents indefinitely. Use the Resolved filter to hide closed ones.

Notifications stopped after a server restart

Boot logs should show alert engine starting. If they don't, check journalctl -u observekit-server (or the container logs) for startup errors.

The most common cause is a SQLite migration failure on a custom build. The error log will say which migration failed.

I need to test what an alert would look like before saving

Use Preview on the rule's authoring form. It runs the rule against the last 24 hours of data and shows what would have fired, without saving anything.

For routing, use the Test routing drawer on the Routing tab — enter a hypothetical label set, see which policies match.

For channels, use Send test on the channel in Settings.

How do I see what was sent to which channel?

Each incident's detail page has a Notification log section: every dispatch attempt with channel, status, and timestamp. The Settings → Notification Channels screen shows recent send activity per channel.

I'm still stuck

Check Settings → System info for the server version. Then file an issue with:

• Server version.
• The rule, routing policy, and channel config (screenshots OK).
• The Incidents detail page if there's an incident.
• The notification log entries.

Most issues are config-shaped, not code-shaped — the answer is usually in one of the screenshots above.