Kubernetes Audit Log Monitoring

Overview

Kubernetes Audit Log Monitoring provides deep visibility into all interactions with the Kubernetes API server. Every request made to the Kubernetes API is recorded, including requests from:

Control plane components (scheduler, controllers)
Node components (kubelet, kube-proxy)
Cluster services
Users interacting through kubectl
Applications calling the Kubernetes API

When enabled, Kubernetes Audit Log Monitoring helps you:

Track all API server activity in the cluster
Detect unauthorized access and privilege escalation
Identify RBAC policy misconfigurations
Monitor resource lifecycle events (create, update, delete)
Investigate security incidents and suspicious activity
Diagnose permission issues and API failures

Requirements

Component	Requirement
Agent	`atatus-agent` Helm chart
Version	4.2.0+
Cluster	Kubernetes / K3s cluster
Access	Node access to Kubernetes audit log files

Enabling Kubernetes Audit Logs

For more information about setting up Kubernetes audit logs, see Kubernetes Auditing. For K8s Selfhosted, configure the K8s server with audit logging enabled. To enable audit logs in Kubernetes:

Audit logs are disabled by default in Kubernetes. To enable them in your API server configuration, specify an audit policy file path:

kube-apiserver
  [...]
  --audit-log-path=/var/log/kubernetes/apiserver/audit.log
  --audit-policy-file=/etc/kubernetes/audit-policies/policy.yaml

Create the policy file at /etc/kubernetes/audit-policies/policy.yaml to specify the types of API requests you want to capture in your audit logs. Audit policy rules are evaluated in order. The API server follows the first matching rule it finds for each type of operation or resource. Example of an audit policy:

Example minimal audit policy:

# /etc/kubernetes/audit-policies/policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
    # do not log requests to the following
    - level: None
      nonResourceURLs:
          - '/healthz*'
          - '/logs'
          - '/metrics'
          - '/swagger*'
          - '/version'

    # limit level to Metadata so token is not included in the spec/status
    - level: Metadata
      omitStages:
          - RequestReceived
      resources:
          - group: authentication.k8s.io
            resources:
                - tokenreviews

    # extended audit of auth delegation
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          - group: authorization.k8s.io
            resources:
                - subjectaccessreviews

    # log changes to pods at RequestResponse level
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          # core API group; add third-party API services and your API services if needed
          - group: ''
            resources: ['pods']
            verbs: ['create', 'patch', 'update', 'delete']

    # log everything else at Metadata level
    - level: Metadata
      omitStages:
          - RequestReceived

This example policy file configures the API server to log at the highest level of detail for certain types of cluster-changing operations (update, patch, create, delete). It also tracks requests to the subjectaccessreviews resource at the highest level to help troubleshoot authentication delegation issues. You may want to reduce the level of verbosity to Metadata for endpoints that contain sensitive data, such as the tokenreviews resource. Datadog also omits the RequestReceived stage from logs. In the last section, for everything that was not explicitly configured by the previous rules, the policy is configured to log at Metadata level. As audit logs might be verbose, you can choose to exclude less critical actions/verbs, such as operations that don’t change the cluster state like list, watch, and get. After updating the configuration, restart the K8s server.

Installing the Atatus Agent Helm Chart

If the Atatus agent is not installed in your Kubernetes cluster, first install the agent by following the Kubernetes installation guide Install Atatus Agent using Helm Once the agent is installed, you can update the Helm chart to enable Kubernetes audit log monitoring. Update Helm Repository Add the Atatus Helm repository if it is not already configured:

helm repo add atatus https://atatus.github.io/helm-charts
helm repo update

Pull the Helm chart locally: Download the chart locally to modify configuration values:

helm pull atatus/atatus-agent --untar
cd atatus-agent

Upgrade the Existing Agent

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace

This upgrades the existing agent deployment and applies the configuration changes required for Kubernetes audit log collection.

Agent Configuration

The Atatus agent must be configured to:

Enable log collection
Mount the Kubernetes audit log directory
Read the audit log file

Enable Log Collection

Edit values.yaml:

atatus:
  logs_enabled: true

Without this setting, file logs will not be shipped.

Mount Kubernetes Audit Logs

Kubernetes audit logs are written on the node filesystem by the API server. The Atatus agent must mount this directory to read and ship the logs. Add the audit log directory as a hostPath volume in values.yaml.

daemonset:
  extraVolumes:
    - name: pointdir
      hostPath:
        path: /opt/atatus-agent/run
        type: DirectoryOrCreate

    - name: auditdir
      hostPath:
        path: /var/log/kubernetes
        type: DirectoryOrCreate

Mount the Directory Inside the Agent Container Mount the host directory into the agent container so the agent can read the audit log file.

daemonset:
  extraVolumeMounts:
    - name: pointdir
      mountPath: /opt/atatus-agent/run

    - name: auditdir
      mountPath: /var/log/kubernetes

This maps the host directory:

/var/log/kubernetes

to the container path:

/var/log/kubernetes

The agent will read audit logs from this mounted directory. Common Audit Log Paths The exact audit log directory depends on the Kubernetes distribution.

Environment	Typical Audit Log Path
Kubernetes (kubeadm)	`/var/log/kubernetes/apiserver/audit.log`
K3s	`/var/lib/rancher/k3s/server/logs/audit.log`

Verify the Log File Exists Before deploying the agent, verify that the audit log file exists on the node:

ls -l /var/log/kubernetes

Example:

audit.log

Update the DaemonSet Template

Edit template/daemonset.yaml: To enable file log collection, the files.yml configuration must be mounted inside the agent container. Update the templates/daemonset.yaml file in the Helm chart to mount the configuration file. Add the following volume mount inside the container volumeMounts section: If the audit log file does not exist, ensure Kubernetes audit logging is enabled on the API server.

- name: atatus-agent-config
  mountPath: /etc/atatus-infra-agent/conf.d/files.d/files.yml
  subPath: files.yml

Configure Log Collection

Configure the agent to collect the audit log file. Edit values.yaml:

daemonset:
  atatusConfig:
    files.yml: |-
      logs:
        - type: file
          paths:
            - /var/log/kubernetes/audit.log
          service: audit
          source: kubernetes.audit

The file path inside the container must match the mounted directory.

K3s Full Example Setup

The following example shows a complete K3s setup without changing the generic Kubernetes configuration above.

1. Create the K3s audit policy file

Create /var/lib/rancher/k3s/server/audit.yaml on the K3s server:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /livez*
      - /readyz*
      - /metrics
      - /version
  - level: Metadata
    resources:
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  - level: RequestResponse
    resources:
      - group: authorization.k8s.io
        resources:
          - subjectaccessreviews
  - level: RequestResponse
    verbs:
      - create
      - update
      - patch
      - delete
    resources:
      - group: ""
        resources:
          - pods
          - secrets
          - configmaps
      - group: rbac.authorization.k8s.io
        resources:
          - roles
          - rolebindings
          - clusterroles
          - clusterrolebindings
  - level: Metadata

2. Update the K3s server config

Edit /etc/rancher/k3s/config.yaml:

kube-apiserver-arg:
  - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml
  - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log
  - audit-log-maxage=30
  - audit-log-maxbackup=10
  - audit-log-maxsize=100

Restart K3s:

sudo systemctl restart k3s

Verify the audit log file exists:

sudo ls -lh /var/lib/rancher/k3s/server/logs/audit.log
sudo tail -n 20 /var/lib/rancher/k3s/server/logs/audit.log

3. K3s-specific `values.yaml` example

Because K3s writes audit logs to /var/lib/rancher/k3s/server/logs/audit.log, use that host path in the Helm values:

atatus:
  logs_enabled: true

daemonset:
  extraVolumes:
    - name: pointdir
      hostPath:
        path: /opt/atatus-agent/run
        type: DirectoryOrCreate
    - name: auditdir
      hostPath:
        path: /var/lib/rancher/k3s/server/logs
        type: DirectoryOrCreate

  extraVolumeMounts:
    - name: pointdir
      mountPath: /opt/atatus-agent/run
    - name: auditdir
      mountPath: /var/log/kubernetes

  atatusConfig:
    files.yml: |-
      logs:
        - type: file
          paths:
            - /var/log/kubernetes/audit.log
          service: audit
          source: kubernetes.audit

4. Update the DaemonSet template for `files.yml`

Add the following mount inside the container volumeMounts section in templates/daemonset.yaml:

- name: atatus-agent-config
  mountPath: /etc/atatus-infra-agent/conf.d/files.d/files.yml
  subPath: files.yml

This ensures the files.yml configuration from values.yaml is mounted into the Atatus agent container.

5. Deploy and verify

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace
kubectl exec -n atatus ds/atatus-agent -- ls -l /var/log/kubernetes/audit.log

If the file exists inside the pod, the agent can collect Kubernetes audit logs from the K3s server.

Deploy the Agent

Deploy or upgrade the Helm chart:

helm upgrade --install atatus-agent . -f values.yaml -n atatus --create-namespace

Verify the agent pod can access the audit log file:

kubectl exec -n atatus ds/atatus-agent -- ls -l /var/log/kubernetes/audit.log

If the file exists and log collection is enabled, the agent will begin shipping Kubernetes audit logs.

Example Security Events

Kubernetes audit logs capture important cluster activities such as:

Event	Description
Pod creation	Detect new containers launched
Pod deletion	Identify potential malicious cleanup
Secret access	Track access to sensitive data
RBAC changes	Monitor role and permission updates
Container exec	Detect interactive container sessions
Anonymous access	Detect unauthenticated API requests

Example audit event:

{
  "verb": "create",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "nginx"
  },
  "user": {
    "username": "system:admin"
  },
  "sourceIPs": ["10.0.0.10"]
}

Viewing Audit Events

Audit events appear in: Security → Cloud SIEM → Kubernetes Audit Logs Each event includes:

User or service account
Resource type and namespace
API operation (create, update, delete)
Source IP address
Request status
Timestamp
Audit stage

These events provide a full audit trail of activity inside your Kubernetes cluster.

​Overview

​Requirements

​Enabling Kubernetes Audit Logs

​Installing the Atatus Agent Helm Chart

​This upgrades the existing agent deployment and applies the configuration changes required for Kubernetes audit log collection.

​Agent Configuration

​Enable Log Collection

​Mount Kubernetes Audit Logs

​Update the DaemonSet Template

​Configure Log Collection

​K3s Full Example Setup

​1. Create the K3s audit policy file

​2. Update the K3s server config

​3. K3s-specific values.yaml example

​4. Update the DaemonSet template for files.yml

​5. Deploy and verify

​Deploy the Agent

​Example Security Events

​Viewing Audit Events