Move the Azure log forwarder to Kubernetes

The Azure quickstart runs the log forwarder as Azure Container Apps jobs in your subscription. If you run a Kubernetes cluster such as AKS, you can run the same jobs there as CronJobs instead, and delete the Container App jobs. Do this after the quickstart has finished with log forwarding selected. Nothing changes in Atatus. The same logs keep arriving, from every region you already collect from, and the jobs continue from where the Container App jobs stopped. After the move, the cluster, the CronJobs and their upgrades are yours to manage. Atatus does not create, update or remove them.

What moves

The quickstart creates these Container App jobs in the resource group atatus-log-forwarder:

Container App job	How many	What it does
`atatus-log-forwarder-<id>`	One	Sends logs from the region you set up first
`atatus-log-reconcile-<id>`	One	Keeps the `atatus-logs` diagnostic setting on your resources, and removes it when you turn log collection off
`atatus-logfwd-<id>`	One for each additional region	Sends logs from that region

Each one becomes a Kubernetes CronJob with the same name, schedule, image and settings. These stay where they are, because the CronJobs use them:

The storage accounts in atatus-log-forwarder. Azure writes your logs there, and the jobs read them and keep their progress there.
The atatus-integration App Registration and its roles.
The diagnostic settings on your resources.

Before you start

You need:

A cluster with Linux amd64 nodes. The forwarder image is not built for arm64.
Outbound access on port 443 from the pods to *.blob.core.windows.net, management.azure.com, login.microsoftonline.com and the Atatus log intake.
az, kubectl and jq. Azure Cloud Shell has all three.
Permission to read and delete the jobs in atatus-log-forwarder (Contributor on that resource group), and to create a namespace, Secrets and CronJobs in the cluster.

Each run needs little capacity: a forwarder run usually finishes in a few seconds.

Step 1: Connect kubectl to your cluster

For AKS:

az aks get-credentials --resource-group <cluster resource group> --name <cluster name>
kubectl get nodes

At least one node must show Ready.

Step 2: Move the jobs

Run this script. For each Container App job it:

Copies the image, schedule, settings and secrets into a Kubernetes Secret and CronJob with the same name, in the namespace atatus.
Parks the Container App job on a once-a-year schedule. The job stops running, but still exists, so you can go back until you delete it in step 4.

RG=atatus-log-forwarder
NS=atatus
PARKED="0 0 1 1 *"

kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -

for JOB in $(az containerapp job list -g "$RG" --query "[].name" -o tsv); do
  SPEC=$(az containerapp job show -n "$JOB" -g "$RG" -o json)
  CRON=$(jq -r '.properties.configuration.scheduleTriggerConfig.cronExpression' <<<"$SPEC")
  if [ "$CRON" = "$PARKED" ]; then echo "$JOB is already parked, skipping"; continue; fi

  IMAGE=$(jq -r '.properties.template.containers[0].image' <<<"$SPEC")
  TIMEOUT=$(jq -r '.properties.configuration.replicaTimeout' <<<"$SPEC")
  SECRETS=$(az containerapp job secret list -n "$JOB" -g "$RG" --show-values -o json)
  echo "$JOB  schedule='$CRON'  image=$IMAGE" | tee -a aca-jobs-before.txt

  # Same variables the Container App job had, secrets included
  ENVFILE=$(mktemp)
  jq -r --argjson s "$SECRETS" '
    .properties.template.containers[0].env[] as $e
    | $e.name + "=" + (if $e.secretRef
        then (first($s[] | select(.name == $e.secretRef)) | .value)
        else ($e.value // "") end)' <<<"$SPEC" > "$ENVFILE"
  kubectl -n "$NS" create secret generic "$JOB" --from-env-file="$ENVFILE" \
    --dry-run=client -o yaml | kubectl apply -f -
  rm -f "$ENVFILE"

  # Park the Container App job before its replacement starts
  az containerapp job update -n "$JOB" -g "$RG" --cron-expression "$PARKED" -o none

  kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: CronJob
metadata:
  name: $JOB
  namespace: $NS
spec:
  schedule: "$CRON"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: $TIMEOUT
      template:
        spec:
          restartPolicy: Never
          nodeSelector:
            kubernetes.io/os: linux
            kubernetes.io/arch: amd64
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          containers:
            - name: forwarder
              image: $IMAGE
              envFrom:
                - secretRef:
                    name: $JOB
              resources:
                requests: { cpu: 100m, memory: 128Mi }
                limits: { cpu: 500m, memory: 1Gi }
              securityContext:
                allowPrivilegeEscalation: false
                capabilities: { drop: ["ALL"] }
EOF
done

kubectl -n "$NS" get cronjobs

az prints a warning that the output contains secret values. That is expected: the script reads them so it can copy them into the Kubernetes Secrets. The last command lists one CronJob for each Container App job you had.

Step 3: Check that the pods work

Do both checks before you delete anything. The pods deliver the logs. Wait two minutes, then print the end of the latest run of each CronJob:

for CJ in $(kubectl -n atatus get cronjobs -o name | cut -d/ -f2); do
  echo "=== $CJ"
  POD=$(kubectl -n atatus get pods --sort-by=.metadata.creationTimestamp -o name | grep "$CJ" | tail -1)
  [ -n "$POD" ] && kubectl -n atatus logs "$POD" --tail=5 || echo "no run yet"
done

A forwarder run ends with a line like this. 0 failure(s) means Atatus accepted the logs:

forwarded 1022 record(s) from 5 blob(s), 1916205 bytes, 0 failure(s), 5 blob(s) seen, backlog 0 bytes

The reconcile job runs every 10 minutes, so it may show no run yet. To run it now:

RECONCILE=$(kubectl -n atatus get cronjobs -o name | grep reconcile | cut -d/ -f2)
kubectl -n atatus create job --from=cronjob/"$RECONCILE" reconcile-check
kubectl -n atatus logs job/reconcile-check -f

It ends with a line that starts with reconciled:. Atatus still receives logs. In Atatus, open the Azure account. The Log Collection tab should show the forwarder as seen in the last few minutes, and Logs should show new Azure records. A finished pod shows Completed, with 0/1 under Ready. That is normal for a pod that has done its work and exited. A real failure shows Error, CrashLoopBackOff or ImagePullBackOff.

Step 4: Delete the Container App jobs

Do this once step 3 is clean. Until you delete them, the parked jobs still run once a year, on 1 January. This deletes only jobs that are parked:

RG=atatus-log-forwarder
for JOB in $(az containerapp job list -g "$RG" \
    --query "[?properties.configuration.scheduleTriggerConfig.cronExpression=='0 0 1 1 *'].name" -o tsv); do
  az containerapp job delete --name "$JOB" --resource-group "$RG" --yes
done

Do not delete the atatus-log-forwarder resource group. It holds the storage accounts the pods read from.

After the move

Settings you change in Atatus. They keep working as before. The jobs ask Atatus what to collect at the start of every run, so turning log collection on or off, and changing regions, resource types, tag filters or subscriptions, takes effect within 10 minutes.
New regions. When you create resources in a region you do not collect from yet, the reconcile job deploys a Container Apps forwarder for that region. To stay on Kubernetes only, either limit log regions on the Log Collection tab in Atatus to the regions you already have, or run the script in step 2 again when a new job appears. It moves only jobs that are not parked.
A job that comes back. Check az containerapp job list -g atatus-log-forwarder a day after step 4. If a job has reappeared, run steps 2 and 4 again.
Secrets. The Kubernetes Secrets in the atatus namespace hold your Atatus license key, the storage account keys and the App Registration client secret. Limit who can read Secrets in that namespace. If you rotate any of these, update the Secret as well.
Upgrades. The CronJobs keep the image tag the Container App jobs had. With the default latest tag, each run uses the newest forwarder image and there is nothing to upgrade.
CPU and memory. Each run requests 0.1 vCPU and 128 MiB, and may use up to 0.5 vCPU and 1 GiB. If the backlog figure in a forwarder’s log grows from run to run, raise the limits:
kubectl -n atatus set resources cronjob <cronjob name> \ --requests=cpu=250m,memory=256Mi --limits=cpu=1,memory=2Gi

Do not run the quickstart again with log forwarding selected. It deploys the Container App jobs again, next to your CronJobs, and two reconcile jobs must not run at the same time. Run it only to go back to Container Apps, after deleting the atatus namespace.

Troubleshooting

What you see	Cause	Fix
Pod status `ImagePullBackOff`	The node cannot pull the image from Docker Hub	Allow outbound access to `docker.io`, or copy the image to a registry the cluster can reach and change the CronJob’s image
Pod stays `Pending`	No Linux amd64 node has room	Add capacity, or lower the requests
Forwarder log shows failures	The pod cannot reach the Atatus intake or the storage account	Check outbound access on port 443
`control: integration is ...; not forwarding this run`	Log collection is off, or the integration is disabled, in Atatus	Switch Enable log collection on
`refused` above zero in the reconcile log, with `could not connect <resource>` lines	Azure rejected a diagnostic setting, for example because the resource already has five	Remove an unused diagnostic setting from that resource
`refused` above zero with no `could not connect` line	The job could not read that resource’s log categories. Nothing was changed on it	No action needed. The job tries again on later runs

Go back to Container Apps

Before step 4, the Container App jobs still exist. aca-jobs-before.txt lists each job’s original schedule. Remove the pods and restore the schedules:

kubectl delete namespace atatus

# Each forwarder job
az containerapp job update --name <forwarder job> --resource-group atatus-log-forwarder --cron-expression "* * * * *"
# The reconcile job
az containerapp job update --name <reconcile job> --resource-group atatus-log-forwarder --cron-expression "*/10 * * * *"

After step 4, delete the namespace as above, then run the quickstart again with log forwarding selected.

Remove log forwarding later

The order matters. The diagnostic settings sit on your resources, and the reconcile job holds the only list of where they are.

In Atatus, delete the integration, or switch Enable log collection off.
Leave the CronJobs running. Within 10 minutes the reconcile job removes the diagnostic settings and the stored logs. Wait for this line in its log:
control: teardown reported complete; Atatus can now remove the integration
If you check the log later and that line has gone, look for a line that contains teardown completed at followed by a time and ends with not reconciling. The reconcile job logs it on every run after teardown, and it means the same thing.
If you deleted the integration, delete the Kubernetes resources and the resource group the same day. A reconcile CronJob left running for more than a day after the integration is deleted can add the diagnostic settings back.
kubectl delete namespace atatus az group delete --name atatus-log-forwarder
If you only switched log collection off, leave everything in place. Collection resumes when you switch it on again.

​What moves

​Before you start

​Step 1: Connect kubectl to your cluster

​Step 2: Move the jobs

​Step 3: Check that the pods work

​Step 4: Delete the Container App jobs

​After the move

​Troubleshooting

​Go back to Container Apps

​Remove log forwarding later

What moves

Before you start

Step 1: Connect kubectl to your cluster

Step 2: Move the jobs

Step 3: Check that the pods work

Step 4: Delete the Container App jobs

After the move

Troubleshooting

Go back to Container Apps

Remove log forwarding later