AWS VPC Flow Logs - Atatus Docs

Overview

VPC Flow Logs record the network connections in and out of your VPC: source and destination address, ports, protocol, byte and packet counts, and whether the traffic was accepted or rejected. Atatus reads them directly from the S3 bucket your flow log already delivers to, using the same agent and the same configuration file as AWS CloudTrail. Only source: differs. Rejected flows are the ones worth watching: a port scan, a misconfigured security group, or blocked exfiltration all look like a burst of REJECT records.

Collection starts from the moment you set it up. Objects already in the bucket are not ingested.

Requirements

Component	Requirement
Agent	`atatus-infra-agent`
Version	4.3.0+
Runs on	One Linux machine that can reach AWS
AWS	A VPC flow log delivering to S3 in the default format
Permissions	Read access to that bucket

The flow log must use AWS’s default format. A custom format has a different column order, and its fields are left empty rather than being guessed at and attached to the wrong columns. Wrong data is worse than missing data in a detection engine.

Step 1: Create the flow log

Skip this if you already deliver flow logs to S3.

aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-ids vpc-0123456789abcdef0 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::my-flow-logs

Objects land under:

AWSLogs/<account-id>/vpcflowlogs/<region>/<yyyy>/<mm>/<dd>/

Flow logs are delivered in batches, so allow up to 10 minutes between the traffic and the object appearing.

Step 2: Create the IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AtatusListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-flow-logs"
    },
    {
      "Sid": "AtatusReadObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-flow-logs/*"
    }
  ]
}

Read-only: the agent never writes to your bucket and never deletes an object. Attach it to the instance role of the machine running the agent, or to an IAM user whose access key you paste into the config. The SQS notification mode described on the CloudTrail page works here too. Add the queue permissions from that page if you prefer it.

Step 3: Configure the agent

Add a stanza to /etc/atatus-infra-agent/conf.d/s3.d/s3.yml. If you already collect CloudTrail, add this alongside the existing entry, since one file holds every AWS log family.

s3:
  - source: vpc_flow
    region: "<region>"
    bucket_arn: "<bucket_arn>"
    bucket_list_prefix: "<bucket_list_prefix>"
    bucket_list_interval: 5m
    number_of_workers: 5

    access_key_id: "<access_key_id>"
    secret_access_key: "<secret_access_key>"

Filled in:

s3:
  - source: vpc_flow
    region: "us-east-1"
    bucket_arn: "arn:aws:s3:::my-flow-logs"
    bucket_list_prefix: "AWSLogs/123456789012/vpcflowlogs/"
    bucket_list_interval: 5m
    number_of_workers: 5

    access_key_id: "AKIAIOSFODNN7EXAMPLE"
    secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Both families in one file:

s3:
  - source: cloudtrail
    region: "us-east-1"
    bucket_arn: "arn:aws:s3:::my-cloudtrail-logs"
    bucket_list_prefix: "AWSLogs/123456789012/CloudTrail/"

  - source: vpc_flow
    region: "us-east-1"
    bucket_arn: "arn:aws:s3:::my-flow-logs"
    bucket_list_prefix: "AWSLogs/123456789012/vpcflowlogs/"

Reading a bucket in another AWS account

Add role_arn and the agent assumes that role, reading with its temporary credentials instead of anything stored on disk:

s3:
  - source: vpc_flow
    region: "us-east-1"
    bucket_arn: "arn:aws:s3:::central-flow-logs"
    bucket_list_prefix: "AWSLogs/999999999999/vpcflowlogs/"

    role_arn: "arn:aws:iam::999999999999:role/AtatusLogReader"
    external_id: "atatus-7f3c1b"

The role needs the read policy above plus a trust policy naming the agent’s own account. See Reading an S3 bucket in another AWS account for the full walkthrough. It is identical for every family, and only source: and the prefix differ.

Check the real key path before setting bucket_list_prefix. A prefix that matches nothing lists zero objects and reports no error, which looks exactly like “no traffic yet”.Confirm with: aws s3 ls s3://my-flow-logs/ --recursive | head -3Leaving it out entirely reads the whole bucket, which always works but is slower on a large one.

Step 4: Restart and verify

sudo service atatus-infra-agent restart

Then open SIEM → Security Events and filter on dataSource: aws_vpc_flow_logs.

What gets collected

Each flow record becomes one security event:

Field	From the flow log
`sourceIp` / `destinationIp`	`srcaddr` / `dstaddr`
`sourcePort` / `destinationPort`	`srcport` / `dstport`
`networkProtocol`	`protocol`, resolved to a name (`6` → `tcp`)
`networkBytesIn` / `networkPacketsIn`	`bytes` / `packets`
`eventAction`	`ACCEPT` or `REJECT`
`eventOutcome`	`success`, or `failure` for a rejected flow
`cloudAccountId`	`account-id`
`logSourceId`	`interface-id` (the ENI)
`eventDateTime`	`end` (when the flow finished)

Severity is low for accepted traffic and medium for rejected traffic, before the detection rules run.

Configuration reference

Setting	Required	Description
`source`	Yes	`vpc_flow`
`region`	Yes	AWS region of the queue or bucket
`queue_url`	Mode A	SQS queue URL. Selects notification mode
`bucket_arn`	Mode B	Bucket ARN. Selects polling mode
`bucket_list_prefix`	No	Restrict polling to a key prefix
`bucket_list_interval`	No	How often to list the bucket. Default `120s`
`number_of_workers`	No	Parallel object readers. Default `5`
`access_key_id`	No	Use with `secret_access_key`
`secret_access_key`	No	Use with `access_key_id`
`role_arn`	No	Assume this role and read with its temporary credentials
`external_id`	No	Shared secret when the role’s trust policy requires one
`start_timestamp`	No	Collect only objects newer than this (RFC3339)
`siem`	No	Set `false` to send to Logs instead of SIEM
`labels`	No	Extra key/value labels on every event

queue_url and bucket_arn are mutually exclusive, so set one, not both.

How it behaves

Only new logs are collected. Collection begins when the agent first starts with this configuration.
Restarts are safe. The agent records which objects it has already sent, so restarting the agent or rebooting the machine never re-sends them.
Volume. Flow logs are far higher-volume than CloudTrail. Start with one VPC before enabling it fleet-wide.

Troubleshooting

Errors appear in the agent log at /var/log/atatus-infra-agent/.

Symptom	Likely cause	Fix
`s3 listing matched no objects`	`bucket_list_prefix` does not match the real key path	Run `aws s3 ls s3://<bucket>/ --recursive \| head` and use the actual prefix
`s3 listing failed`	Missing `s3:ListBucket`, or wrong bucket ARN	Check the IAM policy and the ARN
`AccessDenied` from `sts:AssumeRole`	Trust policy does not name your account, or `external_id` does not match	Check both sides of the trust policy
Events arrive, but ports and byte counts are empty	The flow log uses a custom format	Recreate it with the default format
`every listed object was filtered out`	`start_timestamp` is later than the objects	Remove it, or move it earlier
Events arrive as logs, no detections	`siem: false` is set	Remove it

​Overview

​Requirements

​Step 1: Create the flow log

​Step 2: Create the IAM policy

​Step 3: Configure the agent

​Reading a bucket in another AWS account

​Step 4: Restart and verify

​What gets collected

​Configuration reference

​How it behaves

​Troubleshooting

Overview

Requirements

Step 1: Create the flow log

Step 2: Create the IAM policy

Step 3: Configure the agent

Reading a bucket in another AWS account

Step 4: Restart and verify

What gets collected

Configuration reference

How it behaves

Troubleshooting