ObserveKit Help
Back to app

Concepts

ObserveKit alerts are built around four ideas. If you grasp these, the rest of the system explains itself.

1. Rules detect

A rule is a question ObserveKit asks repeatedly: *is this thing wrong right now?*

A rule has:

  • A condition — a metric threshold, a log pattern, an anomaly. Pick from a template or write your own expression.
  • A scope — which series the rule applies to. Empty scope means "every series".
  • A sustain window — how long the condition must hold before firing. Stops false alarms from short blips.
  • A severity — Critical, Warning, or Info.

When the condition is true for the sustain window, the rule fires an alert. When it falls back below the threshold, the rule resolves.

2. Routing decides

A fired alert is just a label set. Routing policies decide what happens next:

  • Which channels get notified.
  • How long to wait before the first notification.
  • How alerts are grouped into incidents.
  • When to renotify if nobody acknowledges.

Routing is a tree. Alerts walk the tree top-down. The first matching policy wins. You can have child policies that override the parent's defaults.

3. Incidents persist

An incident is a fired alert that's been routed to a channel. Incidents persist across rule re-fires:

  • If the same alert fires again within the reopen window, the existing incident reopens instead of a new one being created.
  • Several alerts can group into one incident — controlled by Group by on the routing policy.
  • Acknowledging an incident silences renotifications until it auto-expires.

The Incidents page lists every incident, open or closed, with full history.

4. Silences suppress

A silence is a temporary mute. Define matchers, set a window, add a comment. Any alert whose labels match the silence — within the window — is suppressed.

Use silences for known maintenance, vendor outages, or noisy alerts you're already investigating.

How they fit together

metric / log lines
       │
       ▼
   ┌───────┐    ┌──────────┐    ┌────────┐
   │ Rule  │ →  │ Routing  │ →  │Channel │
   │  ?    │    │ policy   │    │(email, │
   └───────┘    └──────────┘    │ Slack, │
       │            │           │ Teams, │
       │            ▼           │webhook,│
       │       ┌─────────┐      │PagerDy)│
       │       │Incident │      └────────┘
       │       │ + group │
       │       └─────────┘
       ▼
  ┌─────────┐
  │Silence? │ ← suppresses dispatch
  └─────────┘

Where to next

NextAuthoring rules