17 Prometheus checks

Overview

Two steps are required to start gathering Prometheus data:

an pointing to the appropriate data endpoint, e.g.

There are two Prometheus data preprocessing options:

Prometheus pattern - used in normal items to query Prometheus data
Prometheus to JSON - used in normal items and for low-level discovery. In this case queried Prometheus data are returned in a JSON format.

Bulk processing

Bulk processing is supported for dependent items. To enable caching and indexing, the Prometheus pattern preprocessing must be the first preprocessing step. When Prometheus pattern is first preprocessing step then the parsed Prometheus data is cached and indexed by the first <label>==<value> condition in the Prometheus pattern preprocessing step. This cache is reused when processing other dependent items in this batch. For optimal performance, the first label should be the one with most different values.

If there is other preprocessing to be done before the first step, it should be moved either to the master item or to a new dependent item which would be used as the master item for the dependent items.

Configuration

Enter general dependent item parameters in the configuration form
Go to the Preprocessing tab
Select a Prometheus preprocessing option (Prometheus pattern or Prometheus to JSON)

The following parameters are specific to the Prometheus pattern preprocessing option:

Examples of using parameters

The most common use case is to return the value. To return the value of from:

Result processing - select ‘value’

You may also be interested in the average value of all node_disk_usage_bytes parameters:

Pattern - node_disk_usage_bytes
Result processing - select ‘avg’

While Prometheus supports only numerical data, it is popular to use a workaround that allows to return the relevant textual description as well. This can be accomplished with a filter and specifying the label. So, to return the value of the ‘color’ label from

elasticsearch_cluster_health_status{cluster="elasticsearch",color="green"} 1
elasticsearch_cluster_health_status{cluster="elasticsearch",color="red"} 0

use the following parameters:

Pattern - elasticsearch_cluster_health_status {cluster="elasticsearch"} == 1
Label - specify ‘color’

The filter (based on the numeric value ‘1’) will match the appropriate row, while the label will return the health status description (currently ‘green’; but potentially also ‘red’ or ‘yellow’).

Prometheus to JSON

Data from Prometheus can be used for low-level discovery. In this case data in JSON format are needed and the Prometheus to JSON preprocessing option will return exactly that.

Query language comparison

The following table lists differences and similarities between PromQL and Zabbix Prometheus preprocessing query language.

PromQL instant vector selector		Zabbix Prometheus preprocessing
Differences
Query target	Prometheus server	Plain text in Prometheus exposition format
Returns	Instant vector	Metric or label value (Prometheus pattern) Array of metrics for single value in JSON (Prometheus to JSON)
Label matching operators	=, !=, =~, !~	=, !=, =~, !~
Regular expression used in label or metric name matching	RE2	PCRE
Comparison operators	See	Only == (equal) is supported for value filtering
Similarities
Selecting by metric name that equals string	<metric name> or {name=”<metric name>”}	<metric name> or {name=”<metric name>”}
Selecting by metric name that matches regular expression	{name=~”<regex>”}	{name=~”<regex>”}
Selecting by <label name> value that equals string	{<label name>=”<label value>”,…}	{<label name>=”<label value>”,…}
Selecting by <label name> value that matches regular expression	{<label name>=~”<regex>”,…}	{<label name>=~”<regex>”,…}
Selecting by value that equals string	{name=~”.“} == <value>	{name=~”.“} == <value>