Prometheus

    Istio provides a basic sample installation to quickly get Prometheus up and running:

    This will deploy Prometheus into your cluster. This is intended for demonstration only, and is not tuned for performance or security.

    While the quick-start configuration is well-suited for small clusters and monitoring for short time horizons, it is not suitable for large-scale meshes or monitoring over a period of days or weeks. In particular, the introduced labels can increase metrics cardinality, requiring a large amount of storage. And, when trying to identify trends and differences in traffic over time, access to historical data can be paramount.

    Consult the to get started deploying Prometheus into your environment. See Configuration for more information on configuring Prometheus to scrape Istio deployments.

    In an Istio mesh, each component exposes an endpoint that emits metrics. Prometheus works by scraping these endpoints and collecting the results. This is configured through the which controls settings for which endpoints to query, the port and path to query, TLS settings, and more.

    To gather metrics for the entire mesh, configure Prometheus to scrape:

    1. The control plane ( deployment)
    2. Ingress and Egress gateways
    3. The Envoy sidecar
    4. The user applications (if they expose Prometheus metrics)

    To simplify configuration, Istio has the ability to control scraping entirely by prometheus.io annotations. This allows Istio scraping to work out of the box with standard configurations such as the ones provided by the Helm stable/prometheus charts.

    While prometheus.io annotations are not a core part of Prometheus, they have become the de facto standard to configure scraping.

    This option is enabled by default but can be disabled by passing --set meshConfig.enablePrometheusMerge=false during . When enabled, appropriate prometheus.io annotations will be added to all data plane pods to set up scraping. If these annotations already exist, they will be overwritten. With this option, the Envoy sidecar will merge Istio’s metrics with the application metrics. The merged metrics will be scraped from /stats/prometheus:15020.

    This option exposes all the metrics in plain text.

    This feature may not suit your needs in the following situations:

    • You need to scrape metrics using TLS.
    • Your application exposes metrics with the same names as Istio metrics. For example, your application metrics expose an istio_requests_total metric. This might happen if the application is itself running Envoy.

    If required, this feature can be disabled per workload by adding a prometheus.istio.io/merge-metrics: "false" annotation on a pod.

    • To scrape Istiod stats, the following example job can be added to scrape its http-monitoring port:
    1. - job_name: 'istiod'
    3.   - role: endpoints
    4.     namespaces:
    5.       names:
    6.       - istio-system
    7.   relabel_configs:
    8.   - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    9.     action: keep
    10.     regex: istiod;http-monitoring
    • To scrape Envoy stats, including sidecar proxies and gateway proxies, the following job can be added to scrape ports that end with -envoy-prom:
    • For application stats, if Strict mTLS is not enabled, your existing scraping configuration should still work. Otherwise, Prometheus needs to be configured to .

    TLS settings

    The control plane, gateway, and Envoy sidecar metrics will all be scraped over plaintext. However, the application metrics will follow whatever Istio configuration has been configured for the workload. In particular, if Strict mTLS is enabled, then Prometheus will need to be configured to scrape using Istio certificates.

    One way to provision Istio certificates for Prometheus is by injecting a sidecar which will rotate SDS certificates and output them to a volume that can be shared with Prometheus. However, the sidecar should not intercept requests for Prometheus because the Prometheus’s model of direct endpoint access is incompatible with Istio’s sidecar proxy model.

    To achieve this, configure a cert volume mount on the Prometheus server container:

    1. containers:
    2.     ...
    4.       mountPath: /etc/prom-certs/
    5.       name: istio-certs
    6. volumes:
    7.   - emptyDir:
    8.       medium: Memory
    9.     name: istio-certs

    Then add the following annotations to the Prometheus deployment pod template, and deploy it with . This configures the sidecar to write a certificate to the shared volume, but without configuring traffic redirection:

    Finally, set the scraping job TLS context as follows:

    1. scheme: https
    2. tls_config:
    3.   ca_file: /etc/prom-certs/root-cert.pem
    4.   cert_file: /etc/prom-certs/cert-chain.pem
    5.   key_file: /etc/prom-certs/key.pem
    6.   insecure_skip_verify: true  # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate

    For larger meshes, advanced configuration might help Prometheus scale. See Using Prometheus for production-scale monitoring for more information.