Using OpenEBS as TSDB for Prometheus

    Prometheus is the mostly widely used application for scraping cloud native application metrics. Prometheus and OpenEBS together provide a complete open source stack for monitoring. In this solution, OpenEBS is used as Prometheus TSDB, where all the metrics are permanently stored on local Kubernetes cluster.

    When using OpenEBS as TSDB, following are the advantages:

    • All the data is stored locally and managed natively to Kubernetes

    • No need of externally managed Prometheus storage

    • Prometheus metrics are highly available. When a node fails or rebooted, Prometheus pod is rescheduled onto on of the two other nodes where cStor volume replica is available. The metrics data is rebuilt when the node becomes available

    • Take backup of the Prometheus metrics periodically and back them up to S3 or any object storage so that restoration of the same metrics is possible to the same or any other Kubernetes cluster

    Deployment model

    As shown above, OpenEBS volumes need to be configured with three replicas for high availability. This configuration work fine when the nodes (hence the cStor pool) is deployed across Kubernetes zones.

    1. Install OpenEBS : If OpenEBS is not installed on the Kubernetes already, start by OpenEBS on all or some of the cluster nodes. If OpenEBS is already installed, go to step 2.

    2. Configure cStor Pool : After OpenEBS installation, cStor pool has to be configured. As prometheus TSDB needs high availability of data, OpenEBS cStor volume has to be configured with three replicas. During cStor Pool creation, make sure that the maxPools parameter is set to >=3. If cStor Pool is already configured as required go to Step 4 to create Prometheus StorageClass.

    3. StorageClass is the interface through which most of the OpenEBS storage policies are defined. See Prometheus Storage Class section below.

    4. Configure PVC : Prometheus needs only one volume to store the data. See PVC example spec below.

    5. Persistent volume for Grafana:

      Grafana needs a much smaller persistent storage for storing metadata. Typically the storage class used for Prometheus is reused for Grafana as well. Just construct a new PVC with smaller storage size.

    Reference at

    A live deployment of Prometheus using OpenEBS volumes as highly available TSDB storage can be seen at the website www.openebs.ci

    Deployment YAML spec files for Prometheus and OpenEBS resources are found

    OpenEBS-CI dashboard of Prometheus

    Monitor OpenEBS Volume size

    Monitor cStor Pool size

    As in most cases, cStor pool may not be dedicated to just Prometheus alone. It is recommended to watch the pool capacity and add more disks to the pool before it hits 80% threshold. See cStorPool metrics

    Maintain volume replica quorum during node upgrades

    cStor volume replicas need to be in quorum Prometheus application is deployed as and cStor volume is configured to have . Node reboots may be common during Kubernetes upgrade. Maintain volume replica quorum in such instances. See for more details.

    ,

    Sample YAML specs

    Sample cStor Pool spec

    Prometheus StorageClass

    PVC spec for Prometheus

    PVC spec for Grafana

    See the for Grafana using cStor here.

    Feedback

    Was this page helpful?

    Thanks for the feedback. Open an issue in the if you want to report a problem or suggest an improvement. Engage and get additional help on https://kubernetes.slack.com/messages/openebs/.