Set Up the Agent with Terraform for DocumentDB

This page provisions the agent side of Database Monitoring for Amazon DocumentDB with Terraform: an EC2 host running the Atatus Infrastructure Agent, the TLS material it needs, and the security group rule that lets it reach your cluster.

The same stack can also monitor RDS or Aurora PostgreSQL from that one host. See Set Up the Agent with Terraform for PostgreSQL, and Monitoring both engines below for running them together.

Before you begin

Three ways DocumentDB differs from PostgreSQL

Worth reading before you plan. Each of these fails in a way that does not name its own cause.

Its IAM is a different mechanism

Both engines can use IAM, but not the same way. RDS PostgreSQL mints a short lived rds-db:connect token and presents it as the password, which is why it needs an IAM policy naming a dbuser ARN.

DocumentDB has no such token. The driver signs each handshake itself using the MONGODB-AWS SASL mechanism, so there is no IAM policy to attach. Instead you map the agent's role ARN to a user inside the cluster. Nothing has to be enabled on the cluster or its parameter group first.

The agent connects per instance, not per cluster

A cluster endpoint points at the current primary and a reader endpoint balances across replicas. Either moves the agent to a different node behind its back, and the metrics then describe a host they are not attributed to.

So the agent reaches every instance directly, and the config carries one entry per instance. Terraform expands a cluster into those entries for you, so documentdb_clusters stays a list of clusters rather than a list of hostnames you maintain by hand.

Discovery calls aws docdb describe-db-clusters, because hashicorp/aws ships no aws_docdb_cluster data source and the endpoints cannot be derived from the cluster endpoint by string surgery. That is why the AWS CLI has to be present wherever you run Terraform.

TLS is required and the agent does not ship this CA

DocumentDB requires TLS and presents a certificate from a private Amazon PKI that is in no operating system trust store. Under PostgreSQL managed authentication the agent carries the RDS bundle itself; for MongoDB it does not.

The agent host downloads the bundle at boot and checks that what arrived really is a certificate, because a proxy returning an HTML error page otherwise leaves a file that exists and fails every connection.

Get the Terraform

$ git clone https://github.com/atatus/atatus-database-monitoring-sample.git
$ cd atatus-database-monitoring-sample/terraform/aws/ec2
$ cp terraform.tfvars.example terraform.tfvars

Store the license key

$ aws ssm put-parameter --name /atatus/license_key \
    --type SecureString --value '<YOUR_LICENSE_KEY>'

Configure

A DocumentDB only setup, with databases left empty:

aws_region                  = "ap-south-1"
vpc_id                      = "vpc-0123456789abcdef0"
subnet_id                   = "subnet-0123456789abcdef0"
associate_public_ip_address = true

databases                             = {}
documentdb_use_managed_authentication = true

documentdb_clusters = {
  prod = {
    cluster_identifier = "docdb-prod"
    security_group_id  = "sg-0123456789abcdef0"
    db_name            = "admin"
  }
}

license_key_ssm_parameter_name = "/atatus/license_key"
os_family                      = "amazon-linux"
instance_type                  = "t3.small"

Read your cluster's VPC and security group off the cluster itself:

$ aws docdb describe-db-clusters --db-cluster-identifier <CLUSTER_IDENTIFIER> \
    --query 'DBClusters[0].{SG:VpcSecurityGroups[].VpcSecurityGroupId,SubnetGroup:DBSubnetGroup,Members:DBClusterMembers[].DBInstanceIdentifier}'

Replica sets

A DocumentDB cluster is a replica set: one primary and zero or more replicas. You do not configure the replicas individually. Give Terraform the cluster identifier and it discovers the members, then writes one config entry per instance:

documentdb_clusters = {
  prod = {
    cluster_identifier = "docdb-prod"
    security_group_id  = "sg-0123456789abcdef0"
  }
}

A three node cluster produces three entries, all sharing one cluster_name so they are grouped together in the UI. Check what was resolved:

$ terraform output documentdb_instance_endpoints
$ terraform output monitored_target_count

{
  "prod" = [
    "docdb-prod-1.abc123.ap-south-1.docdb.amazonaws.com:27017",
    "docdb-prod-2.abc123.ap-south-1.docdb.amazonaws.com:27017",
    "docdb-prod-3.abc123.ap-south-1.docdb.amazonaws.com:27017",
  ]
}

Why one entry per member and not one cluster endpoint

A replica set has three kinds of address, and only one of them is stable per node:

If the agent used the cluster endpoint, a failover would silently move it to a different node while the metrics kept the old host's name. Reporting each member separately is what lets you see replication lag, per node load, and a replica falling behind.

After scaling the cluster

Discovery runs at plan time, so adding or removing a replica is picked up by an apply:

$ terraform apply

The agent host is replaced, because the config is written at first boot and the boot script runs only once.

If you pinned instance_endpoints by hand instead of letting discovery run, that list is now stale as well, and it will not follow a scale out. That is the main reason to prefer discovery.

Failover

Nothing needs to change. Every member is already being collected from directly, so when the primary moves, the agent keeps reporting all of them and the roles swap over in the data. The IAM mapping is per cluster, not per node, so it survives a failover untouched.

Monitoring several clusters

documentdb_clusters is a map, and one agent host collects from every entry:

documentdb_clusters = {
  prod = {
    cluster_identifier = "docdb-prod"
    security_group_id  = "sg-0123456789abcdef0"
    labels             = { team = "core" }
  }

  staging = {
    cluster_identifier = "docdb-staging"
    security_group_id  = "sg-0123456789abcdef1"
    cluster_name       = "staging-cluster"
  }
}

Each cluster is expanded into one entry per instance, so a three node cluster and a one node cluster produce four entries in total. Clusters that share a security group and port share one ingress rule.

terraform output monitored_target_count reports how many entries were produced, and documentdb_instance_endpoints shows which hosts they point at.

Create the database user

The agent authenticates as a user that has to exist in the cluster before it starts. There are two ways to create it.

Automatically, at first boot

This is the option worth taking here, because the user's name is the agent's own IAM role ARN, and that role does not exist until Terraform creates it. Doing it on the host removes an apply, read the ARN, create the user, restart sequence.

$ aws ssm put-parameter --name /atatus/docdb_master_password \
    --type SecureString --value '<MASTER_PASSWORD>'

documentdb_clusters = {
  prod = {
    cluster_identifier = "docdb-prod"
    security_group_id  = "sg-0123456789abcdef0"

    bootstrap_master_password_ssm_parameter_name = "/atatus/docdb_master_password"
    bootstrap_master_username                    = "root"
  }
}

The host installs mongosh, connects to the cluster endpoint, which always points at the current primary because createUser is a write, and creates the user. If it already exists, only the missing roles are granted, so a re run never revokes anything granted out of band. The call is retried, because cloud init can win the race against the cluster becoming reachable.

By hand

Connect to the cluster's primary as your master user, after the apply, and use the role ARN from the outputs:

$ terraform output -raw agent_iam_role_arn
$ curl -sSo /tmp/rds-ca.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

use $external
db.createUser({
  user: "arn:aws:iam::<ACCOUNT_ID>:role/<AGENT_ROLE_NAME>",
  mechanisms: ["MONGODB-AWS"],
  roles: [
    { role: "readAnyDatabase", db: "admin" },
    { role: "clusterMonitor",  db: "admin" }
  ]
})

Then restart the agent, because it will have been failing to authenticate until the user existed:

$ sudo systemctl restart atatus-infra-agent

Using a password instead

If you would rather not use IAM, give each cluster its own password parameter and turn managed authentication off:

documentdb_use_managed_authentication = false

documentdb_clusters = {
  prod = {
    cluster_identifier          = "docdb-prod"
    security_group_id           = "sg-0123456789abcdef0"
    password_ssm_parameter_name = "/atatus/docdb_prod_password"
    username                    = "atatus"
  }
}

Two clusters are two separate credentials, so there is no fleet wide password setting. Create the user with the mongo shell:

use admin
db.createUser({
  user: "atatus",
  pwd: "<PASSWORD>",
  roles: [
    { role: "read", db: "admin" },
    { role: "clusterMonitor", db: "admin" }
  ]
})

Apply

$ terraform init
$ terraform plan
$ terraform apply

Verify

$ terraform output monitored_target_count
$ terraform output documentdb_instance_endpoints
$ terraform output -raw ssm_start_session_command
$ terraform output -raw verification_command

To read the boot script's own account, including whether the user was created:

$ sudo journalctl -t atatus-bootstrap --no-pager
$ sudo tail -40 /var/log/cloud-init-output.log

Monitoring both engines from one host

The agent reads conf.d/postgresql.d/ and conf.d/mongodb.d/ independently, so one machine covers both. Add a databases map alongside documentdb_clusters:

use_managed_authentication            = true
documentdb_use_managed_authentication = true

databases = {
  main = {
    identifier        = "prod-pg"
    kind              = "instance"
    security_group_id = "sg-0123456789abcdef1"
  }
}

documentdb_clusters = {
  prod = {
    cluster_identifier = "docdb-prod"
    security_group_id  = "sg-0123456789abcdef0"
  }
}

use_managed_authentication covers PostgreSQL and documentdb_use_managed_authentication covers DocumentDB, and the second defaults to the first, so one setting turns IAM on for both. Set them separately to keep DocumentDB on a password while PostgreSQL uses IAM.

The PostgreSQL side has its own prerequisites, including pg_stat_statements and a database reboot you may need to schedule. See Set Up the Agent with Terraform for PostgreSQL.

Removing it

$ terraform destroy

Your cluster is untouched. The atatus user and the SSM parameters are not Terraform's and stay behind:

use $external
db.dropUser("arn:aws:iam::<ACCOUNT_ID>:role/<AGENT_ROLE_NAME>")

Related pages

• Set Up the Agent with Terraform for PostgreSQL
• Database Monitoring for AWS DocumentDB