Aggregate Monitoring Data of Multiple TiDB Clusters

    Thanos is a high availability solution for Prometheus that simplifies the availability guarantee of Prometheus.

    Thanos provides component as a unified query solution across multiple Prometheus clusters. You can use this feature to aggregate monitoring data of multiple TiDB clusters.

    1. Configure a Thanos Sidecar container for each TidbMonitor.

      A configuration example is as follows.

    2. Deploy the Thanos Query component.

      1. Download the file for Thanos Query deployment:

        1. curl -sl -O https://raw.githubusercontent.com/pingcap/tidb-operator/master/examples/monitor-with-thanos/thanos-query.yaml

      2. Manually modify the --store parameter in the thanos-query.yaml file by updating basic-prometheus:10901 to basic-prometheus.${namespace}:10901.

      3. Execute the kubectl apply command for deployment.

        In the command above, ${thanos_namespace} is the namespace where the Thanos Query component is deployed.

    In Thanos Query, a Prometheus instance corresponds to a store and also corresponds to a TidbMonitor. After deploying Thanos Query, you can provide a uniform query interface for monitoring data through Thanos Query’s API.

    To access the Thanos Query panel, execute the following command, and then access http://127.0.0.1:9090 in your browser:

    1. kubectl port-forward -n ${thanos_namespace} svc/thanos-query 9090

    If you want to access the Thanos Query panel using NodePort or LoadBalancer, refer to the following documents:

    After deploying Thanos Query, to query the monitoring data of multiple TidbMonitors, take the following steps:

    1. In the left navigation bar, select Configuration > Data Sources.
    2. Add or modify a DataSource in the Prometheus type.
    3. Set the URL under HTTP to http://thanos-query.${thanos_namespace}:9090.

    In Thanos Query, a Prometheus instance corresponds to a monitor store and also corresponds to a TidbMonitor. If you need to add, update, or remove a monitor store from the Thanos Query, update the --store configuration of the Thanos Query component, and perform a rolling update to the Thanos Query component.

    To ensure successful configuration, you must first create the S3 bucket. If you choose AWS S3, refer to and AWS documentation - AWS S3 Endpoint List for instructions.

    Thanos Sidecar supports replicating monitoring data to S3 remote storage.

    The configuration of the TidbMonitor CR is as follows:

    2.   thanos:
    3.     baseImage: thanosio/thanos
    4.     version: v0.17.2
    5.     objectStorageConfig:
    6.       key: objectstorage.yaml
    7.       name: thanos-objectstorage

    Meanwhile, you need to create a Secret. The example is as follows:

    Besides aggregating data via Thanos Query, you can also push monitoring data to Thanos using Prometheus’ RemoteWrite feature.

    To enable the RemoteWrite mode, specify the Prometheus RemoteWrite configuration when you create the TidbMonitor CR. For example:

    1. apiVersion: pingcap.com/v1alpha1
    2. kind: TidbMonitor
    3. metadata:
    4.   name: basic
    5. spec:
    6.   clusters:
    7.   - name: basic
    9.     version: v2.27.1
    10.     remoteWrite:
    11.       - url: "http://thanos-receiver:19291/api/v1/receive"
    12.   grafana:
    13.     baseImage: grafana/grafana
    14.     version: 7.5.11
    15.   initializer:
    16.     baseImage: registry.cn-beijing.aliyuncs.com/tidb/tidb-monitor-initializer
    17.     version: v5.4.0
    18.   reloader:
    19.     baseImage: registry.cn-beijing.aliyuncs.com/tidb/tidb-monitor-reloader
    20.     version: v1.0.1
    21.   prometheusReloader:
    22.     baseImage: quay.io/prometheus-operator/prometheus-config-reloader
    24.   imagePullPolicy: IfNotPresent

    After RemoteWrite is enabled, Prometheus pushes the monitoring data to . For more information, refer to the design of Thanos Receiver.