Investigating a problem

The pages in ObserveKit connect to each other. This is how to walk from a symptom to a root cause instead of staring at one page.

How the signals link up

• `trace_id` ties a log line to its span, and a span to every other span in the same request. It's the pivot between Logs and Traces.
• `service_name` ties spans to the Services RED metrics and the Service Map.
• `exception_fingerprint` groups repeated stack traces on the Exceptions page.
• Deployment markers (Deployments) put a "what changed" line on your charts.

Runbook: a latency spike

1. Services → find the service with the raised p99. Open it.
2. Sort its recent traces by duration; open the slowest.
3. In the trace's span tree, find the span with the long bar — that's where the time went (a slow DB call, a downstream service, etc.).
4. Copy the span's trace_id, then in Logs filter trace_id = <id> (or SQL WHERE trace_id = '<id>') to read exactly what that request logged.

Runbook: an error spike

1. Overview → the error-rate donut or the Errors tile → jump to Logs filtered to level = error.
2. If it's an exception, open Exceptions — the fingerprint grouping shows you which stack trace is spiking and how often.
3. From a log line or exception, pivot via trace_id to the trace to see the full request path and which service actually failed.

Runbook: a crash-looping pod

1. Infrastructure → find the pod with climbing restarts.
2. Open its logs (or Logs filtered by pod_name) and read the last lines before each restart.
3. If it's crashing on an exception, the Exceptions page will have the grouped stack trace.

Runbook: did the deploy cause it?

1. Deployments → find the marker near when the symptom started.
2. Compare the error rate / latency just before vs after the marker (the marker overlays the charts).
3. Drill from the marker into Logs for that source and window to see the first failures after the rollout.

Runbook: everything about one request

You have a trace_id (from a log, a trace, or a customer report):

-- the request's full span tree
SELECT name, service_name, duration_ns/1e6 AS ms, status_code
FROM spans WHERE trace_id = '<trace-id>' ORDER BY start_time;

-- every log line the request produced
SELECT timestamp, level, message
FROM logs WHERE trace_id = '<trace-id>' ORDER BY timestamp;

When you're not sure where to start

Ask Kit (the assistant) a plain-language question — "why is checkout slow in the last hour?" — to get pointed at the right service/logs, then drill in with the runbooks above. And if a signal is missing entirely, it's a setup issue, not an investigation one: see Troubleshooting: I see no data.