Alerting rules

Alerting rules are configured in Prometheus in the same way as recording rules.

An example rules file with an alert would be:

The labels clause allows specifying a set of additional labels to be attached to the alert. Any existing conflicting labels will be overwritten. The label values can be templated.

The clause specifies a set of informational labels that can be used to store longer additional information such as alert descriptions or runbook links. The annotation values can be templated.

Templating

Examples:

To manually inspect which alerts are active (pending or firing), navigate to the “Alerts” tab of your Prometheus instance. This will show you the exact label sets for which each defined alert is currently active.

Prometheus’s alerting rules are good at figuring what is broken right now, but they are not a fully-fledged notification solution. Another layer is needed to add summarization, notification rate limiting, silencing and alert dependencies on top of the simple alert definitions. In Prometheus’s ecosystem, the Alertmanager takes on this role. Thus, Prometheus may be configured to periodically send information about alert states to an Alertmanager instance, which then takes care of dispatching the right notifications.
Prometheus can be to automatically discover available Alertmanager instances through its service discovery integrations.