Customizing Agent - Kubernetes Monitoring

The Atatus Kubernetes Agent automatically discovers all namespaces and workloads upon installation. This guide outlines the various configuration options available to help you manage data collection, filter logs, and control pod scheduling across your cluster.

All configurations must be added to your atatus_values.yaml file.

Configuration Options

Use the parameters below to customize the agent's behavior:

General Configuration Example

Here is a comprehensive example demonstrating how to construct your atatus_values.yaml file:

# atatus_values.yaml
atatus:
  license_key: "lic_infra_*************"
  logs_enabled: true
  cluster_name: "my-production-cluster"
  log_level: "info"

  # Include/Exclude by namespace
  container_include: kube:preprod kube:staging
  container_exclude: kube:prod

  # Include/Exclude metrics by container name
  container_include_metrics: name:my-app
  container_exclude_metrics: name:sidecar-proxy

  # Include/Exclude logs by image
  container_include_logs: image:my-app-image
  container_exclude_logs: image:fluentd

  # Log line filtering
  container_include_log_lines:
    - '(?i)error'
  container_exclude_log_lines:
    - '(?i)healthcheck'

daemonset:
  enabled: true
  tolerations:
    - operator: "Exists"

Filter Prefixes

All container_include* and container_exclude* options (except _log_lines) require a prefix that tells the agent what to match against. The prefix is followed by the value to match, with no space after the colon. The value can be an exact string or a regular expression.

Note: In a Kubernetes context, "container" in these option names refers to pods, not individual containers within a pod.

The value after the prefix also supports regular expressions:

# Match all namespaces starting with "dev-"
container_include: kube:^dev-.*

# Match all pods whose name starts with "api"
container_include: name:^api.*

# Match all images starting with "python"
container_include: image:^python.*

To filter by multiple values, separate them with spaces:

container_include: kube:preprod kube:staging

This monitors pods in the preprod and staging namespaces, and excludes everything else.

Container Include & Exclude Filtering

Use container_include and container_exclude to control which containers are monitored for both logs and metrics. You can filter by namespace, container name, or image.

• Filter by namespace — monitor only the preprod and staging namespaces, and exclude the prod namespace:

  container_include: kube:preprod kube:staging
  container_exclude: kube:prod

• Filter by container name — monitor only the container named my-app, and exclude the sidecar-proxy container:

  container_include: name:my-app
  container_exclude: name:sidecar-proxy

• Filter by image — monitor only containers running the nginx image, and exclude those running fluentd:

  container_include: image:nginx
  container_exclude: image:fluentd

Container Metrics Filtering

Use container_include_metrics and container_exclude_metrics to control which containers are monitored for metrics only. Log collection is unaffected.

• Filter by container name — collect metrics only from my-app, and exclude the sidecar-proxy container:

  container_include_metrics: name:my-app
  container_exclude_metrics: name:sidecar-proxy

• Filter by namespace — collect metrics only from the production namespace, and exclude the testing namespace:

  container_include_metrics: kube:production
  container_exclude_metrics: kube:testing

Container Logs Filtering

Use container_include_logs and container_exclude_logs to control which containers are monitored for logs only. Metric collection is unaffected.

• Filter by container name — collect logs only from my-app, and exclude the log-router container:

  container_include_logs: name:my-app
  container_exclude_logs: name:log-router

• Filter by image — collect logs only from containers running my-app-image, and exclude those running fluentd:

  container_include_logs: image:my-app-image
  container_exclude_logs: image:fluentd

Log Line Filtering

You can reduce log volume by filtering individual log lines at the agent level using regular expressions. Unlike the container-level filters above, these options match against the content of each log line, not the container name or namespace.

The (?i) prefix in a regular expression enables case-insensitive matching.

• Collect only log lines containing the word "error" (case-insensitive):

  container_include_log_lines:
    - '(?i)error'

• Drop log lines containing the word "healthcheck" (case-insensitive):

  container_exclude_log_lines:
    - '(?i)healthcheck'

Scheduling the Agent on Tainted Nodes

Kubernetes uses taints to restrict which pods can run on specific nodes (e.g., GPU or system nodes). To allow the Atatus Infrastructure Agent to run on tainted nodes, configure tolerations under the daemonset section in your atatus_values.yaml file.

To run the agent on every active node across your cluster, configure a wildcard toleration using the Exists operator:

daemonset:
  enabled: true
  tolerations:
    - operator: "Exists"

Targeting Specific Node Pools

To target specific node pools instead of bypassing all taints, define selective tolerations.

daemonset:
  enabled: true
  tolerations:
    # System node pools
    - key: "CriticalAddonsOnly"
      operator: "Exists"

    # GPU nodes
    - key: "sku"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

    # Spot or preemptible nodes
    - key: "node.kubernetes.io/instance-type"
      operator: "Equal"
      value: "spot"
      effect: "NoSchedule"

    # Windows nodes (if applicable)
    - key: "node.kubernetes.io/os"
      operator: "Equal"
      value: "windows"
      effect: "NoSchedule"