Overview

Kubernetes Audit Log Monitoring provides deep visibility into all interactions with the Kubernetes API server.

Every request made to the Kubernetes API is recorded, including requests from:

• Control plane components (scheduler, controllers)
• Node components (kubelet, kube-proxy)
• Cluster services
• Users interacting through kubectl
• Applications calling the Kubernetes API

When enabled, Kubernetes Audit Log Monitoring helps you:

• Track all API server activity in the cluster
• Detect unauthorized access and privilege escalation
• Identify RBAC policy misconfigurations
• Monitor resource lifecycle events (create, update, delete)
• Investigate security incidents and suspicious activity
• Diagnose permission issues and API failures

Requirements

Enabling Kubernetes Audit Logs

For more information about setting up Kubernetes audit logs, see Kubernetes Auditing.

For K8s Selfhosted, configure the K8s server with audit logging enabled.

To enable audit logs in Kubernetes:

1. Audit logs are disabled by default in Kubernetes. To enable them in your API server configuration, specify an audit policy file path:

kube-apiserver
  [...]
  --audit-log-path=/var/log/kubernetes/apiserver/audit.log
  --audit-policy-file=/etc/kubernetes/audit-policies/policy.yaml

1. Create the policy file at /etc/kubernetes/audit-policies/policy.yaml to specify the types of API requests you want to capture in your audit logs. Audit policy rules are evaluated in order. The API server follows the first matching rule it finds for each type of operation or resource. Example of an audit policy:

Example minimal audit policy:

# /etc/kubernetes/audit-policies/policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
    # do not log requests to the following
    - level: None
      nonResourceURLs:
          - '/healthz*'
          - '/logs'
          - '/metrics'
          - '/swagger*'
          - '/version'

    # limit level to Metadata so token is not included in the spec/status
    - level: Metadata
      omitStages:
          - RequestReceived
      resources:
          - group: authentication.k8s.io
            resources:
                - tokenreviews

    # extended audit of auth delegation
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          - group: authorization.k8s.io
            resources:
                - subjectaccessreviews

    # log changes to pods at RequestResponse level
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          # core API group; add third-party API services and your API services if needed
          - group: ''
            resources: ['pods']
            verbs: ['create', 'patch', 'update', 'delete']

    # log everything else at Metadata level
    - level: Metadata
      omitStages:
          - RequestReceived

This example policy file configures the API server to log at the highest level of detail for certain types of cluster-changing operations (update, patch, create, delete). It also tracks requests to the subjectaccessreviews resource at the highest level to help troubleshoot authentication delegation issues.

You may want to reduce the level of verbosity to Metadata for endpoints that contain sensitive data, such as the tokenreviews resource. Datadog also omits the RequestReceived stage from logs.

In the last section, for everything that was not explicitly configured by the previous rules, the policy is configured to log at Metadata level. As audit logs might be verbose, you can choose to exclude less critical actions/verbs, such as operations that don’t change the cluster state like list, watch, and get.

After updating the configuration, restart the K8s server.

Installing the Atatus Agent Helm Chart

If the Atatus agent is not installed in your Kubernetes cluster, first install the agent by following the Kubernetes installation guide

Install Atatus Agent using Helm

Once the agent is installed, you can update the Helm chart to enable Kubernetes audit log monitoring.

Update Helm Repository

Add the Atatus Helm repository if it is not already configured:

helm repo add atatus https://atatus.github.io/helm-charts
helm repo update

Pull the Helm chart locally:

Download the chart locally to modify configuration values:

helm pull atatus/atatus-agent --untar
cd atatus-agent

Upgrade the Existing Agent

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace

This upgrades the existing agent deployment and applies the configuration changes required for Kubernetes audit log collection.

Agent Configuration

The Atatus agent must be configured to:

1. Enable log collection
2. Mount the Kubernetes audit log directory
3. Read the audit log file

Enable Log Collection

Edit values.yaml:

atatus:
  logs_enabled: true

Without this setting, file logs will not be shipped.

Mount Kubernetes Audit Logs

Kubernetes audit logs are written on the node filesystem by the API server. The Atatus agent must mount this directory to read and ship the logs.

Add the audit log directory as a hostPath volume in values.yaml.

daemonset:
  extraVolumes:
    - name: pointdir
      hostPath:
        path: /opt/atatus-agent/run
        type: DirectoryOrCreate

    - name: auditdir
      hostPath:
        path: /var/log/kubernetes
        type: DirectoryOrCreate

Mount the Directory Inside the Agent Container

Mount the host directory into the agent container so the agent can read the audit log file.

daemonset:
  extraVolumeMounts:
    - name: pointdir
      mountPath: /opt/atatus-agent/run

    - name: auditdir
      mountPath: /var/log/kubernetes

This maps the host directory:

/var/log/kubernetes

to the container path:

/var/log/kubernetes

The agent will read audit logs from this mounted directory.

Common Audit Log Paths

The exact audit log directory depends on the Kubernetes distribution.

Verify the Log File Exists

Before deploying the agent, verify that the audit log file exists on the node:

ls -l /var/log/kubernetes

Example: audit.log

Update the DaemonSet Template

Edit template/daemonset.yaml:

To enable file log collection, the files.yml configuration must be mounted inside the agent container.

Update the templates/daemonset.yaml file in the Helm chart to mount the configuration file.

Add the following volume mount inside the container volumeMounts section:

If the audit log file does not exist, ensure Kubernetes audit logging is enabled on the API server.

- name: atatus-agent-config
  mountPath: /etc/atatus-infra-agent/conf.d/files.d/files.yml
  subPath: files.yml

Configure Log Collection

Configure the agent to collect the audit log file.

Edit values.yaml:

daemonset:
  atatusConfig:
    files.yml: |-
      logs:
        - type: file
          paths:
            - /var/log/kubernetes/audit.log
          service: audit
          source: kubernetes.audit

The file path inside the container must match the mounted directory.

K3s Full Example Setup

The following example shows a complete K3s setup without changing the generic Kubernetes configuration above.

1. Create the K3s audit policy file

Create /var/lib/rancher/k3s/server/audit.yaml on the K3s server:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /livez*
      - /readyz*
      - /metrics
      - /version
  - level: Metadata
    resources:
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  - level: RequestResponse
    resources:
      - group: authorization.k8s.io
        resources:
          - subjectaccessreviews
  - level: RequestResponse
    verbs:
      - create
      - update
      - patch
      - delete
    resources:
      - group: ""
        resources:
          - pods
          - secrets
          - configmaps
      - group: rbac.authorization.k8s.io
        resources:
          - roles
          - rolebindings
          - clusterroles
          - clusterrolebindings
  - level: Metadata

2. Update the K3s server config

Edit /etc/rancher/k3s/config.yaml:

kube-apiserver-arg:
  - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml
  - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log
  - audit-log-maxage=30
  - audit-log-maxbackup=10
  - audit-log-maxsize=100

Restart K3s:

sudo systemctl restart k3s

Verify the audit log file exists:

sudo ls -lh /var/lib/rancher/k3s/server/logs/audit.log
sudo tail -n 20 /var/lib/rancher/k3s/server/logs/audit.log

3. K3s-specific values.yaml example

Because K3s writes audit logs to /var/lib/rancher/k3s/server/logs/audit.log, use that host path in the Helm values:

atatus:
  logs_enabled: true

daemonset:
  extraVolumes:
    - name: pointdir
      hostPath:
        path: /opt/atatus-agent/run
        type: DirectoryOrCreate
    - name: auditdir
      hostPath:
        path: /var/lib/rancher/k3s/server/logs
        type: DirectoryOrCreate

  extraVolumeMounts:
    - name: pointdir
      mountPath: /opt/atatus-agent/run
    - name: auditdir
      mountPath: /var/log/kubernetes

  atatusConfig:
    files.yml: |-
      logs:
        - type: file
          paths:
            - /var/log/kubernetes/audit.log
          service: audit
          source: kubernetes.audit

4. Update the DaemonSet template for files.yml

Add the following mount inside the container volumeMounts section in templates/daemonset.yaml:

- name: atatus-agent-config
  mountPath: /etc/atatus-infra-agent/conf.d/files.d/files.yml
  subPath: files.yml

This ensures the files.yml configuration from values.yaml is mounted into the Atatus agent container.

5. Deploy and verify

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace
kubectl exec -n atatus ds/atatus-agent -- ls -l /var/log/kubernetes/audit.log

If the file exists inside the pod, the agent can collect Kubernetes audit logs from the K3s server.

Deploy the Agent

Deploy or upgrade the Helm chart:

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace

Verify the agent pod can access the audit log file:

kubectl exec -n atatus ds/atatus-agent -- ls -l /var/log/kubernetes/audit.log

If the file exists and log collection is enabled, the agent will begin shipping Kubernetes audit logs.

Example Security Events

Kubernetes audit logs capture important cluster activities such as:

Example audit event:

{
  "verb": "create",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "nginx"
  },
  "user": {
    "username": "system:admin"
  },
  "sourceIPs": ["10.0.0.10"]
}

Viewing Audit Events

Audit events appear in:

Security → Cloud SIEM → Kubernetes Audit Logs

Each event includes:

• User or service account
• Resource type and namespace
• API operation (create, update, delete)
• Source IP address
• Request status
• Timestamp
• Audit stage

These events provide a full audit trail of activity inside your Kubernetes cluster.