Traffic Metrics

You add metrics to a mesh configuration, or to an individual Dataplane configuration if you need, for example, to override the default metrics port that’s already in use on the specified machine.

Kuma provides full integration with Prometheus:

  • Each proxy can expose its metrics in format.
  • Because metrics are part of the mesh configuration, Prometheus can automatically find every proxy in the mesh.

To collect metrics from Kuma, you need to first expose metrics from proxies and then configure Prometheus to collect them.

To expose metrics from every proxy in the mesh, configure the Mesh resource:

which is a convenient shortcut for

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   metrics:
  7.     enabledBackend: prometheus-1
  8.     backends:
  9.     - name: prometheus-1
  10.       type: prometheus
  11.       conf:
  12.         skipMTLS: false
  13.         port: 5670
  14.         path: /metrics
  15.         tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
  16.           kuma.io/service: dataplane-metrics
  1. type: Mesh
  2. name: default
  3. metrics:
  4.   enabledBackend: prometheus-1
  5.   backends:
  6.   - name: prometheus-1
  7.     type: prometheus
  8.     conf:
  9.       skipMTLS: true # by default mTLS metrics are also protected by mTLS. Scraping metrics with mTLS without transparent proxy is not supported at the moment.

which is a convenient shortcut for

  1. type: Mesh
  2. name: default
  3. metrics:
  4.   backends:
  5.   - name: prometheus-1
  6.     type: prometheus
  7.     conf:
  8.       skipMTLS: true
  9.       port: 5670
  10.       path: /metrics
  11.       tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
  12.         kuma.io/service: dataplane-metrics

This tells Kuma to configure every proxy in the default mesh to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

The metrics endpoint is forwarded to the standard Envoy and supports the same query parameters. You can pass the filter query parameter to limit the results to metrics whose names match a given regular expression. By default all available metrics are returned.

Override Prometheus settings per data plane proxy

To override Mesh-wide defaults for a particular Pod, use -specific annotations:

  • prometheus.metrics.kuma.io/port - to override Mesh-wide default port
  • prometheus.metrics.kuma.io/path - to override Mesh-wide default path

For example:

Proxies for this Pod expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

To override Mesh-wide defaults on a particular machine, configure the Dataplane resource:

  1. type: Dataplane
  2. mesh: default
  3. name: example
  4. metrics:
  5.   type: prometheus
  6.   conf:
  7.     skipMTLS: true
  8.     port: 1234
  9.     path: /non-standard-path

This proxy exposes an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Configure Prometheus

In Prometheus version 2.29 and later, you can add Kuma metrics to your prometheus.yml:

  1. scrape_configs:
  2.     - job_name: 'kuma-dataplanes'
  3.       scrape_interval: "5s"
  4.       kuma_sd_configs:
  5.       - server: "http://kuma-control-plane.kuma-system.svc:5676"

For more information, see the Prometheus documentationTraffic Metrics - 图2 (opens new window).

For earlier versions of Prometheus, Kuma provides the kuma-prometheus-sd tool, which runs alongside your Prometheus instance. This tool fetches a list of current data plane proxies from the Kuma control plane and saves the list in Prometheus-compatible format to a file on disk. Prometheus watches for changes to the file and updates its scraping configuration accordingly.

You can run kumactl install metrics | kubectl apply -f - to deploy configured Prometheus with Grafana.

If you’ve already deployed Prometheus, you can use to bring Kuma metrics to your main Prometheus cluster.

  1. Run kuma-prometheus-sd, for example:

    1. kuma-prometheus-sd run \
    2.   --cp-address=grpcs://kuma-control-plane.internal:5676 \
    3.   --output-file=/var/run/kuma-prometheus-sd/kuma.file_sd.json

  2. Configure Prometheus to read from the file you just saved. For example, add the following snippet to prometheus.yml:

    then run:

    1. prometheus --config.file=prometheus.yml

Check the Targets page in the Prometheus dashboard. You should see a list of data plane proxies from your mesh. For example:

A screenshot of Targets page on Prometheus UI

Kuma lets you expose proxy metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the mesh for this feature to work, which is the default deployment model when kumactl install metrics is used on Kubernetes.

Make sure that mTLS is enabled in the mesh.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   mtls:
  7.     enabledBackend: ca-1
  8.     backends:
  9.     - name: ca-1
  10.       type: builtin
  12.     backends:
  13.     - name: prometheus-1
  14.       type: prometheus
  15.       conf:
  16.         port: 5670
  17.         path: /metrics
  18.         skipMTLS: false
  19.         tags: # tags that can be referred in Traffic Permission  
  20.           kuma.io/service: dataplane-metrics

Allow the traffic from Grafana to Prometheus Server and from Prometheus Server to data plane proxy metrics and for other Prometheus components:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: TrafficPermission
  3. mesh: default
  4. metadata:
  5.   name: metrics-permissions
  6. spec:
  7.   sources:
  8.     - match:
  9.        kuma.io/service: prometheus-server_kuma-metrics_svc_80
  10.   destinations:
  11.     - match:
  12.        kuma.io/service: dataplane-metrics
  13.     - match:
  14.        kuma.io/service: "prometheus-alertmanager_kuma-metrics_svc_80"
  15.     - match:
  16.        kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_80"
  17.     - match:
  18.        kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_81"
  19.     - match:
  20.        kuma.io/service: "prometheus-pushgateway_kuma-metrics_svc_9091"
  21. ---
  22. apiVersion: kuma.io/v1alpha1
  23. kind: TrafficPermission
  24. mesh: default
  25. metadata:
  26.   name: grafana-to-prometheus
  27. spec:
  28.    sources:
  29.    - match:
  30.       kuma.io/service: "grafana_kuma-metrics_svc_80"
  31.    destinations:
  32.       kuma.io/service: "prometheus-server_kuma-metrics_svc_80"

In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies.

Use standard prometheus.io annotations on Pod or Service:

Use the Discovery Service of your choice (opens new window). In the future Kuma will help to expose metrics in more native way.

To consume paths protected by mTLS, you need Traffic Permission that lets Prometheus consume applications.

Kuma ships with four default dashboards that are available to import from the Grafana Labs repositoryTraffic Metrics - 图6 (opens new window).

This dashboards lets you investigate the status of a single dataplane in the mesh.

Kuma Dataplane dashboard Kuma Dataplane dashboard

Kuma Mesh

This dashboard lets you investigate the aggregated statistics of a single mesh.

This dashboard lets you investigate aggregated statistics from dataplanes of given source service to dataplanes of given destination service.

Kuma Service to Service dashboard

Kuma CP

This dashboard lets you investigate statistics of the control plane.

Kuma CP dashboard Kuma CP dashboard