Prometheus 2.0 migration guide

    The format of the Prometheus command line flags has changed. Instead of a single dash, all flags now use a double dash. Common flags (, --web.listen-address and --web.external-url) are still the same but beyond that, almost all the storage-related flags have been removed.

    Some notable flags which have been removed:

    • -alertmanager.url In Prometheus 2.0, the command line flags for configuring a static Alertmanager URL have been removed. Alertmanager must now be discovered via service discovery, see Alertmanager service discovery.

    • -log.format In Prometheus 2.0 logs can only be streamed to standard error.

    • -query.staleness-delta has been renamed to --query.lookback-delta; Prometheus 2.0 introduces a new mechanism for handling staleness, see .

    • -storage.local.* Prometheus 2.0 introduces a new storage engine, as such all flags relating to the old engine have been removed. For information on the new engine, see Storage.

    • -storage.remote.* Prometheus 2.0 has removed the already deprecated remote storage flags, and will fail to start if they are supplied. To write to InfluxDB, Graphite, or OpenTSDB use the relevant storage adapter.

    Alertmanager service discovery

    Would be replaced with the following in the prometheus.yml config file:

    1. alerting:
    2.   alertmanagers:
    3.   - static_configs:
    4.     - targets:

    You can also use all the usual Prometheus service discovery integrations and relabeling in your Alertmanager configuration. This snippet instructs Prometheus to search for Kubernetes pods, in the default namespace, with the label name: alertmanager and with a non-empty port.

    1. alerting:
    2.   alertmanagers:
    3.   - kubernetes_sd_configs:
    4.       - role: pod
    6.       ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    7.     bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    8.     relabel_configs:
    9.     - source_labels: [__meta_kubernetes_pod_label_name]
    10.       regex: alertmanager
    11.       action: keep
    12.     - source_labels: [__meta_kubernetes_namespace]
    13.       regex: default
    14.       action: keep
    15.     - source_labels: [__meta_kubernetes_pod_container_port_number]
    16.       regex:
    17.       action: drop

    The format for configuring alerting and recording rules has been changed to YAML. An example of a recording rule and alert in the old format:

    Would look like this:

    1. - name: example.rules
    2.   rules:
    3.   - record: job:request_duration_seconds:histogram_quantile99
    4.     expr: histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m]))
    5.       BY (le, job))
    6.   - alert: FrontendRequestLatency
    7.     expr: job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1
    9.     annotations:
    10.       summary: High frontend request latency

    To help with the change, the promtool tool has a mode to automate the rules conversion. Given a .rules file, it will output a .rules.yml file in the new format. For example:

    1. $ promtool update rules example.rules

    Note that you will need to use promtool from 2.0, not 1.8.

    Storage

    The data format in Prometheus 2.0 has completely changed and is not backwards compatible with 1.8. To retain access to your historic monitoring data we recommend you run a non-scraping Prometheus instance running at least version 1.8.1 in parallel with your Prometheus 2.0 instance, and have the new server read existing data from the old one via the remote read protocol.

    Prometheus 2.0 can then be started (on the same machine) with the following flags:

    1. $ ./prometheus-2.0.0.linux-amd64/prometheus --config.file prometheus.yml

    Where prometheus.yml contains in addition to your full existing configuration, the stanza:

    1. remote_read:
    2.   - url: "http://localhost:9094/api/v1/read"

    The following features have been removed from PromQL:

    • drop_common_labels function - the without aggregation modifier should be used instead.
    • keep_common aggregation modifier - the by modifier should be used instead.
    • count_scalar function - use cases are better handled by absent() or correct propagation of labels in operations.

    See for more details.

    Miscellaneous

    The Prometheus Docker image is now built to . If you want the Prometheus UI/API to listen on a low port number (say, port 80), you’ll need to override it. For Kubernetes, you would use the following YAML:

    See Configure a Security Context for a Pod or Container for more details.

    If you’re using Docker, then the following snippet would be used:

      Prometheus lifecycle