Monitoring Multicluster Istio with Prometheus

    Our recommendation for multicluster monitoring of Istio with Prometheus is built upon the foundation of Prometheus . Prometheus instances that are deployed locally to each cluster by Istio act as initial collectors that then federate up to a production mesh-wide Prometheus instance. That mesh-wide Prometheus can either live outside of the mesh (external), or in one of the clusters within the mesh.

    Follow the multicluster installation section to setup your Istio clusters in one of the supported . For the purpose of this guide, any of those approaches will work, with the following caveat:

    Ensure that a cluster-local Istio Prometheus instance is installed in each cluster.

    Individual Istio deployment of Prometheus in each cluster is required to form the basis of cross-cluster monitoring by way of federation to a production-ready instance of Prometheus that runs externally or in one of the clusters.

    Validate that you have an instance of Prometheus running in each cluster:

    There are several reasons why you may want to have a Prometheus instance running outside of your Istio deployment. Perhaps you want long-term monitoring disjoint from the cluster being monitored. Perhaps you want to monitor multiple separate meshes in a single place. Or maybe you have other motivations. Whatever your reason is, you’ll need some special configurations to make it all work.

    This guide demonstrates connectivity to cluster-local Prometheus instances, but does not address security considerations. For production use, secure access to each Prometheus endpoint with HTTPS. In addition, take precautions, such as using an internal load-balancer instead of a public endpoint and the appropriate configuration of firewall rules.

    Istio provides a way to expose cluster services externally via . You can configure an ingress gateway for the cluster-local Prometheus, providing external connectivity to the in-cluster Prometheus endpoint.

    For each cluster, follow the appropriate instructions from the Remotely Accessing Telemetry Addons task. Also note that you SHOULD establish secure (HTTPS) access.

    Next, configure your external Prometheus instance to access the cluster-local Prometheus instances using a configuration like the following (replacing the ingress domain and cluster name):

    Notes:

    • No authentication to the Prometheus endpoint(s) is provided. This means that anyone can query your cluster-local Prometheus instances. This may not be desirable.

    Production Prometheus on an in-mesh cluster

    If you prefer to run the production Prometheus in one of the clusters, you need to establish connectivity from it to the other cluster-local Prometheus instances in the mesh.

    This is really just a variation of the configuration for external federation. In this case the configuration on the cluster running the production Prometheus is different from the configuration for remote cluster Prometheus scraping.

    In-mesh Production Prometheus for monitoring multicluster Istio

    Configure your production Prometheus to access both of the local and remote Prometheus instances.

    First execute the following command:

    Then add configurations for the remote clusters (replacing the ingress domain and cluster name for each cluster) and add one configuration for the local cluster: