Infrastructure Alerts

Overview

Infrastructure alerts let you monitor the health and performance of your servers, processes, network interfaces, and filesystems. Rules are evaluated on a cron schedule against data stored in ClickHouse. Each rule targets individual hosts, so alerts fire independently per hostname.

Available rule types

Available metrics

System metrics

Storage metrics

Process metrics

Network metrics

Infrastructure checks

Host not reporting

Enter thresholds in the unit shown. For memory metrics, enter the value in GB or MB as shown. For CPU and disk metrics, enter the percentage value directly (e.g., 80 for 80%).

Targets

Infrastructure alerts target resources by hostname. Each alert rule fires independently per host, so a single rule can produce separate incidents for different servers.

Filters

Rules support filtering by hostname and by metric-specific dimensions through the filterConditions field.

Available filter operators

Evaluation logic

Cron-based evaluation

Alert rules are evaluated on a recurring schedule. At each evaluation cycle, the system queries ClickHouse for metric data within the configured duration window.

Duration

The evaluation window can be set to one of the following durations:

• 5 minutes
• 10 minutes
• 15 minutes
• 30 minutes
• 60 minutes

Time functions

Threshold operators

Special rule: host not reporting

The Host Not Reporting rule type uses a different evaluation approach. It checks if the most recent event time for a host is older than the start of the evaluation window. If lastEventTime < startTime, the host is considered stale and the alert fires.

This means the alert triggers when a host has not reported any data for longer than the configured duration.

Special rule: checks failing

The Checks Failing rule type monitors infrastructure health checks. Data is grouped by hostname, check name, and IP address. Within each 1-minute bucket, the system evaluates the check status. If a check reports as down, it is considered failing.

Examples

Example 1: CPU usage alert

Alert when any host's CPU usage exceeds 90% for 5 minutes.

This rule evaluates CPU usage across all hosts. If a host sustains CPU usage above 90% for the full 5-minute window, a critical incident is opened. A warning is raised at 80%.

Example 2: Host not reporting

Alert when a host stops sending data for more than 10 minutes.

At each evaluation cycle, the system checks the last event time for every known host. If a host's last event is more than 10 minutes ago, an incident is created for that host.

Example 3: Disk space alert

Alert when filesystem usage exceeds 85% on any host.

This rule monitors disk usage on hosts with "prod" in their hostname. If any evaluation point within the 15-minute window shows usage above 85%, a critical incident is opened. The any time function means even a brief spike triggers the alert.