Traffic Metrics

A user can enable traffic metrics by editing a Mesh resource and providing the desired Mesh-wide configuration. If necessary, metrics configuration can be customized for each Dataplane individually, e.g. to override the default metrics port that might be already in use on that particular machine.

Out-of-the-box, Kuma provides full integration with Prometheus:

  • if enabled, every dataplane will expose its metrics in Prometheus format
  • furthemore, Kuma will make sure that Prometheus can automatically find every dataplane in the mesh

To collect metrics from Kuma, you need to first expose metrics from Dataplanes and then configure Prometheus to collect them.

To expose Prometheus metrics from every dataplane in the mesh, configure a Mesh resource as follows:

which is a convenient shortcut for

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   metrics:
  7.     enabledBackend: prometheus-1
  8.     backends:
  9.     - name: prometheus-1
  10.       type: prometheus
  11.       conf:
  12.         skipMTLS: false
  13.         port: 5670
  14.         path: /metrics
  15.         tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
  16.           kuma.io/service: dataplane-metrics
  1. type: Mesh
  2. name: default
  3. metrics:
  4.   enabledBackend: prometheus-1
  5.   backends:
  6.   - name: prometheus-1
  7.     type: prometheus
  8.     conf:
  9.       skipMTLS: true # by default mTLS metrics are also protected by mTLS. Scraping metrics with mTLS without transparent proxy is not supported at the moment.

which is a convenient shortcut for

  1. type: Mesh
  2. name: default
  3. metrics:
  4.   enabledBackend: prometheus-1
  5.   backends:
  6.   - name: prometheus-1
  7.     type: prometheus
  8.     conf:
  9.       port: 5670
  10.       path: /metrics
  11.       tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
  12.         kuma.io/service: dataplane-metrics

Both snippets from above instruct Kuma to configure every dataplane in the mesh default to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

Override Prometheus settings per Dataplane

To override Mesh-wide defaults for a particular Pod, use Kuma-specific annotations:

  • prometheus.metrics.kuma.io/port - to override Mesh-wide default port
  • prometheus.metrics.kuma.io/path - to override -wide default path

E.g.,

As a result, dataplane for this particular Pod will expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

To override Mesh-wide defaults on a particular machine, configure Dataplane resource as follows:

  1. type: Dataplane
  2. mesh: default
  3. name: example
  4. metrics:
  5.   type: prometheus
  6.   conf:
  7.     skipMTLS: true
  8.     port: 1234
  9.     path: /non-standard-path

As a result, this particular dataplane will expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Configure Prometheus

Although dataplane metrics are now exposed, Prometheus doesn’t know anything about it just yet.

Use kumactl install metrics | kubectl apply -f - to deploy configured Prometheus with Grafana.

If you’ve got Prometheus deployment already, you can use Prometheus federation (opens new window) to bring Kuma metrics to your main Prometheus cluster.

First, you need to run kuma-prometheus-sd, e.g. by using the following command:

  1. kuma-prometheus-sd run \
  2.   --cp-address=grpcs://kuma-control-plane.internal:5676 \
  3.   --output-file=/var/run/kuma-prometheus-sd/kuma.file_sd.json

The above configuration tells kuma-prometheus-sd to talk to Kuma Control Plane at and save the list of dataplanes to /var/run/kuma-prometheus-sd/kuma.file_sd.json.

Then, you need to set up Prometheus to read from that file, e.g. by using prometheus.yml config with the following contents:

  1. scrape_configs:
  2. - job_name: 'kuma-dataplanes'
  3.   scrape_interval: 15s
  4.   file_sd_configs:
  5.   - files:
  6.     - /var/run/kuma-prometheus-sd/kuma.file_sd.json

and running

Now, if you check Targets page on Prometheus UI, you should see a list of dataplanes from your mesh, e.g.

A screenshot of Targets page on Prometheus UI

Kuma lets you expose Dataplane metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the Mesh for this feature to work, which is the default deployment model when kumactl install metrics is used on Kubernetes.

Make sure that mTLS is enabled in the Mesh.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   mtls:
  7.     enabledBackend: ca-1
  8.     backends:
  9.     - name: ca-1
  10.       type: builtin
  11.   metrics:
  12.     enabledBackend: prometheus-1
  13.     backends:
  15.       type: prometheus
  16.       conf:
  17.         port: 5670
  18.         path: /metrics
  19.         skipMTLS: false
  20.         tags: # tags that can be referred in Traffic Permission  
  21.           kuma.io/service: dataplane-metrics

Allow the traffic from Grafana to Prometheus Server and from Prometheus Server to Dataplane metrics and for other Prometheus components:

  1. apiVersion: kuma.io/v1alpha1
  2. mesh: default
  3. metadata:
  4.   name: metrics-permissions
  5. spec:
  6.   sources:
  7.     - match:
  8.        kuma.io/service: prometheus-server_kuma-metrics_svc_80
  9.   destinations:
  10.     - match:
  11.        kuma.io/service: dataplane-metrics
  12.     - match:
  13.        kuma.io/service: "prometheus-alertmanager_kuma-metrics_svc_80"
  14.     - match:
  15.        kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_80"
  16.     - match:
  17.        kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_81"
  18.     - match:
  19.        kuma.io/service: "prometheus-pushgateway_kuma-metrics_svc_9091"
  20. ---
  21. apiVersion: kuma.io/v1alpha1
  22. kind: TrafficPermission
  23. mesh: default
  24. metadata:
  25.   name: grafana-to-prometheus
  26. spec:
  27.    sources:
  28.    - match:
  29.       kuma.io/service: "grafana_kuma-metrics_svc_80"
  30.    destinations:
  31.    - match:
  32.       kuma.io/service: "prometheus-server_kuma-metrics_svc_80"

This feature requires transparent proxy, therefore for now it’s not available in Universal for now.

In addition to exposing metrics from Dataplane, you may want to expose metrics from application next to Kuma DP.

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   namespace: kuma-example
  5.   name: kuma-tcp-echo
  6. spec:
  7.   ...
  8.   template:
  9.     metadata:
  10.       ...
  11.       annotations:
  12.         prometheus.io/scrape: "true"
  13.         prometheus.io/port: "1234"
  14.         prometheus.io/path: "/non-standard-path"
  15.     spec:

Use Discovery Service of your choice (opens new window). In the future Kuma will help to expose metrics in more native way.

Remember that in order to consume paths protected by mTLS, you need Traffic Permission that lets Prometheus consume applications.

Kuma ships with 4 default dashboards that are available to import from .

This dashboards lets you investigate the status of a single dataplane in the mesh.

Kuma Dataplane dashboard Kuma Dataplane dashboard

Kuma Mesh

This dashboard lets you investigate the aggregated statistics of a single mesh.

This dashboard lets you investigate aggregated statistics from dataplanes of given source service to dataplanes of given destination service.

Kuma Service to Service dashboard

Kuma CP

This dashboard lets you investigate statistics of the control plane.

Kuma CP dashboard Kuma CP dashboard

This dashboard lets you investigate aggregated statistics for each service.