OpsNexus
DocsInfrastructure

Monitoring & alerts

See CPU, memory and disk from your agents, and raise alerts on servers, websites and pipelines with rules checked every minute.

On this page

Monitoring & alerts has three pages in the sidebar:

  • Servers: live CPU, memory and disk for every host running the OpsNexus agent, plus summary tiles.
  • Alerts: alerts that are firing, acknowledged or resolved.
  • Alert rules: the conditions that raise alerts.

Alert rules and alerts belong to the current project. Alerts are sent out through Notifications.

Where the data comes from

Target Source
Servers and agents The OpsNexus agent. It sends a heartbeat every 15 seconds with CPU usage, memory used and total, disk used and total for the root disk, network counters and uptime.
Websites and domains The uptime checks on Domains & SEO.
Pipelines Builds run by pipelines in this project.

There is no separate monitoring agent. The host list is the list of build agents visible to the project: the project's own agents and agents shared with every project. To monitor a server, install the agent on it.

An agent that sends no heartbeat for AGENT_OFFLINE_SECONDS (default 120) is marked offline. Agent metrics are kept for METRIC_RETENTION_DAYS (default 30).

ini
AGENT_OFFLINE_SECONDS=120
METRIC_RETENTION_DAYS=30

The Servers page

The tiles at the top show:

Tile Shows
Open alerts Firing and acknowledged alerts, broken down by severity.
Monitored hosts Hosts with the agent, and how many are reporting.
Domains up Domains that are up, out of all monitored domains.
Alert rules The number of rules in this project.

Below, each host has a card with its name, operating system, when it was last seen, its status, and meters for CPU, Memory and Disk from its latest heartbeat. Meters turn amber at 80% and red at 90%. The page refreshes every 15 seconds.

Alert rules

A rule watches one metric, compares it to a threshold, and raises one alert per target that breaches it. Rules are checked every minute by the background worker.

Rule fields

Field Meaning
Name Shown in the alert and in notifications.
Watch The metric. See the tables below.
Operator >, >=, < or <=.
Threshold The value to compare against, in the metric's unit.
Window (min) 0 to 1440 minutes. How far back to look. 0 means check the current value.
Severity Critical, High, Medium, Low or Info. The alert and its notification get this severity.

Each rule also has an On switch to turn it off without deleting it.

Agents & servers

Metric Unit How the window is used Notification event
CPU usage % Average over the window (at least 1 minute). cpu.high
Memory usage % Average over the window. memory.high
Disk usage % Average over the window. disk.high
Minutes since last heartbeat min Not used. The current gap since the last heartbeat. server.down

Agents that are disabled or in maintenance are skipped.

Websites & domains

Metric Unit How the window is used Notification event
Domain down 1 or 0 1 when every check in the window failed. domain.down
Average response time ms Average of checks in the window. domain.slow
Days until SSL certificate expires days Not used. ssl.expiring
Days until domain registration expires days Not used. domain.expiring

Only enabled domains in the project are checked. Checks recorded as "unknown" (OpsNexus itself was offline) are ignored.

Pipelines & builds

Metric Unit How the window is used Notification event
Failed builds in the window count Failed builds started in the window. build.failed
Build success rate in the window % Succeeded out of finished builds started in the window. build.unstable
Longest build in the window min Longest finished build in the window. build.slow
Minutes a build has waited in the queue min Not used. The longest current wait. build.stuck

Each pipeline is a separate target.

System rules and custom rules

Every new project starts with the recommended system rules:

Rule Condition Window Severity
High CPU CPU usage > 80% 5 min High
High memory Memory usage > 85% 10 min High
Disk almost full Disk usage > 90% 5 min High
Agent not reporting Minutes since last heartbeat > 5 Now Critical
Website down Domain down >= 1 2 min Critical
Slow website Average response time > 2000 ms 10 min Medium
SSL expiring Days until SSL expires < 14 Now Medium
Domain expiring Days until registration expires < 30 Now Medium
Pipeline failing repeatedly Failed builds >= 3 60 min High
Low build success rate Build success rate < 80% 24 h Medium
Slow builds Longest build > 30 min 60 min Medium
Build stuck in queue Queue wait > 15 min Now High

On the Alert rules page, switch between System rules and Custom rules. Rules are grouped into Agents & servers, Websites & domains and Pipelines & builds.

  • You can edit a system rule's threshold, window and severity, or delete it. A deleted system rule shows as Not added; click Add to bring it back, or Add all N missing to restore every missing one.
  • A custom rule never replaces a system rule. A custom "CPU > 95%" runs alongside the system "High CPU".

Create a custom rule

  1. Open Monitoring & alerts > Alert rules.
  2. Click New custom rule.
  3. Enter a Name, pick what to Watch, and set Operator, Threshold, Window (min) and Severity.
  4. Click Save.

To check every rule right away instead of waiting for the next minute, click Evaluate now. It reports how many alerts fired and resolved.

Alerts

An alert opens when a target breaches a rule. There is at most one open alert per rule and target.

Status Meaning
firing The condition is met and nobody has acknowledged it.
acknowledged Someone clicked Acknowledge. The alert stays open.
resolved The condition cleared, or someone clicked Resolve.

The Alerts page filters by Open, Resolved or All, and shows severity, message, target, start time and status.

Auto-resolve

Each evaluation updates the value on open alerts. When the condition no longer holds, the alert is resolved automatically and an alert.resolved notification is sent, including how long the alert lasted.

Pipeline alerts also resolve when the pipeline drops out of the data, for example when there are no failed builds left in the window.

Note

If you resolve an alert by hand while the condition still holds, the next evaluation opens a new alert.

Routing alerts to the right people

When an alert fires, OpsNexus sends the rule's event (see the tables above) with the rule's severity, tagged with the rule's project. A notification goes to:

  • the project's own notification rules that include the event, and
  • notification rules shared with every project that include the event,

as long as the alert's severity is at or above the notification rule's Minimum severity. Without a matching notification rule, the alert only shows in OpsNexus. See Notifications.

The notification card lists the target, metric, current value, threshold, window and rule name, and links to the Alerts page.

Other alerts

Some problems are sent as notifications directly, without alert rules:

  • Kubernetes: cluster down or back, nodes not ready, failing or pending pods, unavailable workloads and HPAs at maximum, for clusters with alerts turned on. See Kubernetes.
  • Cloud budgets: cost.alert when spending crosses a budget threshold. See Cost & budgets.
  • Build agents: agent.connected and agent.disconnected.

These don't appear on the Alerts page.

Permissions

Monitoring uses the Monitoring & alerts module in roles.

Access level Allows
read See servers, alerts and rules. Acknowledge alerts.
modify Create, edit, turn off and delete rules. Add system rules. Evaluate now. Resolve alerts.

Rule changes and alert acknowledgements and resolutions are written to the audit log.