Agent configuration: filtering and tuning

Wherever the ObserveKit agent runs, all of its runtime behavior is controlled by a single YAML file. On Kubernetes it lives as a ConfigMap named observekit-agent in the observekit namespace; on Docker hosts and Linux servers it sits on disk at /etc/observekit/agent.yaml.

This page covers the operator-editable parts — what to write to drop noisy namespaces, silence specific log levels, exclude sidecars, tell a Linux host which log files to tail, and otherwise control the data the agent ships. Everything else in the file (server endpoint, API key, source ID, WAL location, batching) is set at install time and you generally don't touch it.

Most of this page is about the `filters:` block, which subtracts. On a Linux server the important block is `host:`, which *adds* — there is nothing to filter until you have told the agent what to collect.

> Why filter at the agent and not in the UI? Anything the agent drops never crosses the network, never lands in MinIO, and never enters ClickHouse. UI-side filters hide data from your view but do not save bandwidth, storage, or query time. Aggressive agent-side filtering is the single biggest lever for cost and reliability.

Editing on a running cluster

kubectl edit configmap observekit-agent -n observekit
kubectl rollout restart daemonset/observekit-agent -n observekit

The agent reads its config at startup only. A rollout restart is required for any edit to take effect. On a Docker host, edit the file directly and systemctl restart observekit-agent (or docker restart observekit-agent if you run it as a container).

The filters: block

This is the section you'll be editing. Default values shipped with every new install:

filters:
  exclude_namespaces:
    - kube-system
  include_namespaces: []
  exclude_levels: []
  exclude_labels:
    observekit/ignore: "true"

Every other field below is supported but starts empty. All filters compose with AND semantics — an entry survives only if it passes every filter that's set.

By namespace

filters:
  include_namespaces: ["payments", "checkout"]   # whitelist mode — only collect these
  exclude_namespaces: ["kube-system", "istio-system", "logging"]

If include_namespaces is set, the agent ignores everything outside that list (even namespaces not on exclude_namespaces). Leave include_namespaces empty (the default) to collect from everywhere except whatever's on the exclude list.

By log level

filters:
  include_levels: ["error", "warn"]   # only ship error + warn
  exclude_levels: ["debug", "trace"]  # drop debug + trace, ship the rest

The agent parses level from the structured log when it can. Lines whose level cannot be parsed always pass through — exclude_levels will never silently hide an unparsed line. If a level appears in both lists it is DROPPED: the include list is applied first, then the exclude list, so exclude wins on overlap.

By pod label

filters:
  include_labels:
    team: payments              # only ship pods that have team=payments
  exclude_labels:
    observekit/ignore: "true"   # opt-out label any team can set on their own pod
    app.kubernetes.io/name: linkerd-proxy

The default observekit/ignore: "true" is intentional — it lets any team opt their pods out of collection without filing a ticket. Add other entries here to drop pods you don't own.

By container name

filters:
  include_container_names: ["app", "api", "worker"]
  exclude_container_names: ["istio-proxy", "linkerd-proxy", "envoy-sidecar", "vault-agent"]

This is the most surgical filter for sidecar noise. Service-mesh proxies, vault sidecars, and other infra containers often generate as much volume as your application — and you almost never read those logs. Drop them by name here.

By message content (regex)

filters:
  exclude_messages:
    - "health.?check"          # any case, with or without separator
    - "GET /healthz"
    - "ping pong"
  include_messages:
    - "panic:|fatal:|stacktrace"

These are Go regexp/syntax patterns applied to the full assembled log message after multi-line stitching. Invalid regex fails the agent at startup, so a typo will be obvious immediately. Useful for two cases:

• Silencing a known-noisy logger when the app can't be redeployed to fix it at source.
• Forcing include-only mode on a specific signature when you're chasing a specific incident.

A concrete recipe — typical production cluster

This is the configuration we recommend for a steady-state production cluster running a service-mesh:

filters:
  exclude_namespaces:
    - kube-system
    - kube-public
    - kube-node-lease
    - istio-system        # control plane chatter
    - linkerd             # control plane chatter
    - cert-manager        # noisy at renewal
  exclude_container_names:
    - istio-proxy
    - linkerd-proxy
    - envoy-sidecar
  exclude_labels:
    observekit/ignore: "true"
  exclude_messages:
    - "^GET /healthz"
    - "^GET /metrics"
    - "kube-probe/"       # default kubelet probe UA in access logs

In typical clusters this removes 40–70% of the raw log volume without losing anything you'd actually look at.

Tuning shipping behavior (advanced)

These don't live in the filters: block but they're worth knowing because they affect cost and reliability. Defaults are conservative; raise the batch size on high-volume clusters.

collection:
  buffer_size: "50MB"        # on-disk WAL ring buffer
  wal_dir: "/var/lib/observekit/wal"
  batch_interval: 5s         # how often to flush to the server
  batch_max_size: "1MB"      # cap per ship batch
  compression: "zstd"        # leave on; saves ~80% on the wire

Larger batch_max_size and batch_interval mean fewer, bigger HTTP requests — which is gentler on ClickHouse's merger. A cluster sending a steady 1k logs/sec is happier at batch_max_size: 4MB and batch_interval: 10s than at the defaults.

Verifying filters took effect

After editing the ConfigMap and rolling out:

kubectl logs -n observekit -l app=observekit-agent --tail=50 | grep -i filter

You should see lines like filter: dropped 8214 entries in namespace=kube-system and filter: dropped 412 entries by container_name=istio-proxy as soon as the agent starts. If you see your supposedly-dropped namespace still ingesting in the UI's Log Explorer after ~30 seconds, double-check the YAML indentation — a mis-indented key is silently ignored as an unknown field, so the filter you thought you wrote is simply not there.

A genuinely malformed file does not ship everything — it stops the agent. A YAML syntax error, an invalid include_messages / exclude_messages regex, or an unrecognised mode all fail at startup rather than being skipped. That is the safer direction, but it means a bad edit shows up as an agent that will not start (systemctl status observekit-agent, or a CrashLoopBackOff on Kubernetes) rather than as missing data.

Host mode: Linux servers and VMs

On a bare server the agent runs in mode: host and everything it collects is configured under a host: block. See Install on a Linux server or VM for the install itself.

The difference from Kubernetes and Docker is worth stating plainly: there is no discovery. On a container host the agent enumerates containers and starts shipping. On a Linux server nothing identifies your applications, so the agent collects host metrics and otherwise stays quiet until you name the files.

mode: host

host:
  metrics:
    enabled: true
    interval: 30s
    filesystems:
      exclude_fstypes: [tmpfs, devtmpfs, overlay, squashfs]
      include_mountpoints: []      # empty = every filesystem that survives the excludes

  log_files:
    - name: nginx-access
      path: /var/log/nginx/access.log
      labels:
        app: nginx
        stream: access
      multiline:
        enabled: false             # one request per line — never join these

    - name: app
      path: /var/log/myapp/*.log   # globs are re-scanned, so new files appear
      labels:
        app: myapp

  journald:
    enabled: true
    units: [ssh.service, cron.service]   # empty = every unit
    since: 1h                            # unset defaults to 5m

  syslog:
    enabled: false
    paths: [/var/log/syslog, /var/log/auth.log]

  max_log_files: 100

log_files: is a list, and name is required

Each entry gets its own name (emitted as a label and unique across the list), its own labels, and its own multiline rule. Two entries sharing a name would silently merge two unrelated streams under one identity, so the agent rejects it.

Per-file multiline is the reason this is a list rather than a map of paths. An nginx access.log joined with a Java stack-trace pattern glues independent requests into one corrupted record. One server routinely runs both kinds of file, so no single global rule is correct for all of them — set multiline per source, and omit it to inherit the global collection.multiline.

Host metrics use the same names as Kubernetes

host.metrics emits the same node_* and disk_* metric names the Kubernetes node collector uses. Dashboards and alert expressions written for clusters work on a Linux source with no changes.

exclude_fstypes matters more than it looks: pseudo-filesystems are not real disks, and leaving them in means a "disk almost full" alert fires on volumes that cannot fill.

syslog: is off by default, deliberately

auth.log and syslog routinely contain credentials, session details, and other material an operator may not intend to ship off-host. Opting in is a decision you make explicitly, not a default you inherit.

journald.since bounds the first read

An agent starting on a host with months of journal would ship all of it if nothing bounded the first read — a large, surprising ingest bill on day one. Unset, this defaults to 5 minutes, which is deliberately conservative: a restarting agent picks up roughly where it left off without replaying history.

Raise it only when you deliberately want backfill, and size the raise to what you are willing to ingest — since: 24h on a chatty host is a day of journal arriving at once.

max_log_files

Caps how many files are tailed at once across every source, so a wide glob cannot exhaust the process's file descriptors. Leave it unset for the collector's default. A negative value disables the cap — write that deliberately, and own the fd limit.

Applying a host config

sudo systemctl restart observekit-agent

Configs pushed from the UI apply themselves; see Restart behaviour for why the unit must use Restart=always.

What the agent does *not* filter

A few things flow regardless of filters: — by design:

• Heartbeats and version reports to /api/v1/heartbeat. Used to populate the Sources page and the agent-version pill.
• Cluster-level metrics (kube_pod_status_phase, kube_node_status_condition, deployment replicas, etc.). These are cardinality-controlled, not log-line-controlled, and the alert engine depends on them.
• TLS certificate posture for ingress / service certs (used by the Certificate Expiring alert).

If you need to exclude one of these too, it's a server-side change, not an agent config edit. Open a ticket.

When in doubt

Start with the recipe above, deploy it, watch the Cost page for a day, and tighten further if a particular namespace or container still dominates the volume. Filtering is iterative — every team has a different definition of "noise".