ObserveKit Help
Back to app

Agent configuration: filtering and tuning

When you install the ObserveKit agent on a Kubernetes cluster or Docker host, all of its runtime behavior is controlled by a single YAML file. On Kubernetes this file lives as a ConfigMap named observekit-agent in the observekit namespace; on Docker hosts it sits on disk at /etc/observekit/agent.yaml.

This page covers the operator-editable parts — what to put in the ConfigMap to drop noisy namespaces, silence specific log levels, exclude sidecars, and otherwise trim the data the agent ships. Everything else in the file (server endpoint, API key, source ID, WAL location, batching) is set at install time and you generally don't touch it.

> Why filter at the agent and not in the UI? Anything the agent drops never crosses the network, never lands in MinIO, and never enters ClickHouse. UI-side filters hide data from your view but do not save bandwidth, storage, or query time. Aggressive agent-side filtering is the single biggest lever for cost and reliability.

Editing on a running cluster

kubectl edit configmap observekit-agent -n observekit
kubectl rollout restart daemonset/observekit-agent -n observekit

The agent reads its config at startup only. A rollout restart is required for any edit to take effect. On a Docker host, edit the file directly and systemctl restart observekit-agent (or docker restart observekit-agent if you run it as a container).

The filters: block

This is the section you'll be editing. Default values shipped with every new install:

filters:
  exclude_namespaces:
    - kube-system
  include_namespaces: []
  exclude_levels: []
  exclude_labels:
    observekit/ignore: "true"

Every other field below is supported but starts empty. All filters compose with AND semantics — an entry survives only if it passes every filter that's set.

By namespace

filters:
  include_namespaces: ["payments", "checkout"]   # whitelist mode — only collect these
  exclude_namespaces: ["kube-system", "istio-system", "logging"]

If include_namespaces is set, the agent ignores everything outside that list (even namespaces not on exclude_namespaces). Leave include_namespaces empty (the default) to collect from everywhere except whatever's on the exclude list.

By log level

filters:
  include_levels: ["error", "warn"]   # only ship error + warn
  exclude_levels: ["debug", "trace"]  # drop debug + trace, ship the rest

The agent parses level from the structured log when it can. Lines whose level cannot be parsed always pass through — exclude_levels will never silently hide an unparsed line. If both are set, include_levels wins.

By pod label

filters:
  include_labels:
    team: payments              # only ship pods that have team=payments
  exclude_labels:
    observekit/ignore: "true"   # opt-out label any team can set on their own pod
    app.kubernetes.io/name: linkerd-proxy

The default observekit/ignore: "true" is intentional — it lets any team opt their pods out of collection without filing a ticket. Add other entries here to drop pods you don't own.

By container name

filters:
  include_container_names: ["app", "api", "worker"]
  exclude_container_names: ["istio-proxy", "linkerd-proxy", "envoy-sidecar", "vault-agent"]

This is the most surgical filter for sidecar noise. Service-mesh proxies, vault sidecars, and other infra containers often generate as much volume as your application — and you almost never read those logs. Drop them by name here.

By message content (regex)

filters:
  exclude_messages:
    - "health.?check"          # any case, with or without separator
    - "GET /healthz"
    - "ping pong"
  include_messages:
    - "panic:|fatal:|stacktrace"

These are Go regexp/syntax patterns applied to the full assembled log message after multi-line stitching. Invalid regex fails the agent at startup, so a typo will be obvious immediately. Useful for two cases:

  • Silencing a known-noisy logger when the app can't be redeployed to fix it at source.
  • Forcing include-only mode on a specific signature when you're chasing a specific incident.

A concrete recipe — typical production cluster

This is the configuration we recommend for a steady-state production cluster running a service-mesh:

filters:
  exclude_namespaces:
    - kube-system
    - kube-public
    - kube-node-lease
    - istio-system        # control plane chatter
    - linkerd             # control plane chatter
    - cert-manager        # noisy at renewal
  exclude_container_names:
    - istio-proxy
    - linkerd-proxy
    - envoy-sidecar
  exclude_labels:
    observekit/ignore: "true"
  exclude_messages:
    - "^GET /healthz"
    - "^GET /metrics"
    - "kube-probe/"       # default kubelet probe UA in access logs

In typical clusters this removes 40–70% of the raw log volume without losing anything you'd actually look at.

Tuning shipping behavior (advanced)

These don't live in the filters: block but they're worth knowing because they affect cost and reliability. Defaults are conservative; raise the batch size on high-volume clusters.

collection:
  buffer_size: "50MB"        # on-disk WAL ring buffer
  wal_dir: "/var/lib/observekit/wal"
  batch_interval: 5s         # how often to flush to the server
  batch_max_size: "1MB"      # cap per ship batch
  compression: "zstd"        # leave on; saves ~80% on the wire

Larger batch_max_size and batch_interval mean fewer, bigger HTTP requests — which is gentler on ClickHouse's merger. A cluster sending a steady 1k logs/sec is happier at batch_max_size: 4MB and batch_interval: 10s than at the defaults.

Verifying filters took effect

After editing the ConfigMap and rolling out:

kubectl logs -n observekit -l app=observekit-agent --tail=50 | grep -i filter

You should see lines like filter: dropped 8214 entries in namespace=kube-system and filter: dropped 412 entries by container_name=istio-proxy as soon as the agent starts. If you see your supposedly-dropped namespace still ingesting in the UI's Log Explorer after ~30 seconds, double-check the YAML for indentation issues (the agent treats a parse error as "no filters" and ships everything by default).

What the agent does *not* filter

A few things flow regardless of filters: — by design:

  • Heartbeats and version reports to /api/v1/heartbeat. Used to populate the Sources page and the agent-version pill.
  • Cluster-level metrics (kube_pod_status_phase, kube_node_status_condition, deployment replicas, etc.). These are cardinality-controlled, not log-line-controlled, and the alert engine depends on them.
  • TLS certificate posture for ingress / service certs (used by the Certificate Expiring alert).

If you need to exclude one of these too, it's a server-side change, not an agent config edit. Open a ticket.

When in doubt

Start with the recipe above, deploy it, watch the Cost page for a day, and tighten further if a particular namespace or container still dominates the volume. Filtering is iterative — every team has a different definition of "noise".

PreviousSource detail