Classifying Metrics Based on Request or Response

    Counting the number of review requests must account for the unbounded element . GET /reviews/1 followed by GET /reviews/2 should count as two requests to get reviews.

    Istio lets you create classification rules using the that groups requests into a fixed number of logical operations. For example, you can create an operation named GetReviews, which is a common way to identify operations using the Open API Spec operationId. This information is injected into request processing as istio_operationId attribute with value equal to GetReviews. You can use the attribute as a dimension in Istio standard metrics. Similarly, you can track metrics based on other operations like ListReviews and CreateReviews.

    For more information, see the .

    Istio uses the Envoy proxy to generate metrics and provides its configuration in the EnvoyFilter at manifests/charts/istio-control/istio-discovery/templates/telemetryv2_1.10.yaml. As a result, writing classification rules involves adding attributes to the EnvoyFilter.

    You can classify requests based on their type, for example ListReview, GetReview, CreateReview.

    1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin to the EnvoyFilter. It also creates an attribute, istio_operationId and populates it with values for the categories to count as metrics.

      This configuration is service-specific since request paths are typically service-specific.

      1. apiVersion: networking.istio.io/v1alpha3
      2. kind: EnvoyFilter
      3. metadata:
      4.   name: istio-attributegen-filter
      5. spec:
      6.   workloadSelector:
      7.     labels:
      8.       app: reviews
      9.   configPatches:
      10.   - applyTo: HTTP_FILTER
      11.     match:
      12.       context: SIDECAR_INBOUND
      13.       proxy:
      14.         proxyVersion: '1\.9.*'
      15.       listener:
      16.         filterChain:
      17.           filter:
      18.             name: "envoy.http_connection_manager"
      19.             subFilter:
      20.               name: "istio.stats"
      21.     patch:
      22.       operation: INSERT_BEFORE
      23.       value:
      24.         name: istio.attributegen
      25.         typed_config:
      26.           "@type": type.googleapis.com/udpa.type.v1.TypedStruct
      27.           type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
      28.           value:
      29.             config:
      30.               configuration:
      31.                 "@type": type.googleapis.com/google.protobuf.StringValue
      32.                 value: |
      33.                   {
      34.                     "attributes": [
      35.                       {
      36.                         "output_attribute": "istio_operationId",
      37.                         "match": [
      38.                           {
      39.                             "value": "ListReviews",
      40.                             "condition": "request.url_path == '/reviews' && request.method == 'GET'"
      41.                           },
      42.                           {
      43.                             "value": "GetReview",
      44.                             "condition": "request.url_path.matches('^/reviews/[[:alnum:]]*$') && request.method == 'GET'"
      45.                           },
      46.                           {
      47.                             "value": "CreateReview",
      48.                             "condition": "request.url_path == '/reviews/' && request.method == 'POST'"
      49.                           }
      50.                         ]
      51.                       }
      52.                     ]
      53.                   }
      55.                 runtime: envoy.wasm.runtime.null
      56.                   local: { inline_string: "envoy.wasm.attributegen" }

    2. Apply your changes using the following command:

      1. $ kubectl -n istio-system apply -f attribute_gen_service.yaml

    3. Find the stats-filter-1.10 EnvoyFilter resource from the istio-system namespace, using the following command:

      1. $ kubectl -n istio-system get envoyfilter | grep ^stats-filter-1.10
      2. stats-filter-1.10                    2d
      1. $ kubectl -n istio-system get envoyfilter stats-filter-1.10 -o yaml > stats-filter-1.10.yaml

    5. Open stats-filter-1.10.yaml with a text editor and locate the name: istio.stats extension configuration. Update it to map request_operation dimension in the requests_total standard metric to istio_operationId attribute. The updated configuration file section should look like the following.

    6. Save stats-filter-1.10.yaml and then apply the configuration using the following command:

      1. $ kubectl -n istio-system apply -f stats-filter-1.10.yaml

    7. Add the following configuration to the mesh config. This results in the addition of the request_operation as a new dimension to the istio_requests_total metric. Without it, a new metric with the name envoy_request_operation___somevalue___istio_requests_total is created.

      1. meshConfig:
      2.   defaultConfig:
      3.     extraStatTags:
      4.     - request_operation

    8. Generate metrics by sending traffic to your application.

    9. After the changes take effect, visit Prometheus and look for the new or changed dimensions, for example istio_requests_total.

    You can classify responses using a similar process as requests. Do note that the response_code dimension already exists by default. The example below will change how it is populated.

    1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin to the EnvoyFilter and generates the istio_responseClass attribute used by the stats plugin.

      This example classifies various responses, such as grouping all response codes in the 200 range as a 2xx dimension.

      1. apiVersion: networking.istio.io/v1alpha3
      2. kind: EnvoyFilter
      3. metadata:
      4.   name: istio-attributegen-filter
      5. spec:
      6.   workloadSelector:
      7.     labels:
      8.       app: productpage
      9.   configPatches:
      10.   - applyTo: HTTP_FILTER
      11.     match:
      12.       context: SIDECAR_INBOUND
      13.       proxy:
      14.         proxyVersion: '1\.9.*'
      15.       listener:
      16.         filterChain:
      17.           filter:
      18.             name: "envoy.http_connection_manager"
      19.             subFilter:
      20.               name: "istio.stats"
      21.     patch:
      22.       operation: INSERT_BEFORE
      23.       value:
      24.         name: istio.attributegen
      25.         typed_config:
      26.           "@type": type.googleapis.com/udpa.type.v1.TypedStruct
      27.           type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
      28.           value:
      29.             config:
      30.               configuration:
      31.                 "@type": type.googleapis.com/google.protobuf.StringValue
      32.                 value: |
      33.                   {
      34.                     "attributes": [
      35.                       {
      36.                         "output_attribute": "istio_responseClass",
      37.                         "match": [
      39.                             "value": "2xx",
      40.                             "condition": "response.code >= 200 && response.code <= 299"
      41.                           },
      42.                             "value": "3xx",
      43.                             "condition": "response.code >= 300 && response.code <= 399"
      44.                           },
      45.                           {
      46.                             "value": "404",
      47.                             "condition": "response.code == 404"
      48.                           },
      49.                           {
      50.                             "value": "429",
      51.                             "condition": "response.code == 429"
      52.                           },
      53.                           {
      54.                             "value": "503",
      55.                             "condition": "response.code == 503"
      56.                           },
      57.                           {
      58.                             "value": "5xx",
      59.                             "condition": "response.code >= 500 && response.code <= 599"
      60.                           },
      61.                           {
      62.                             "value": "4xx",
      63.                             "condition": "response.code >= 400 && response.code <= 499"
      64.                           }
      65.                         ]
      66.                       }
      67.                     ]
      68.                   }
      69.               vm_config:
      70.                 runtime: envoy.wasm.runtime.null
      71.                 code:
      72.                   local: { inline_string: "envoy.wasm.attributegen" }

    2. Apply your changes using the following command:

      1. $ kubectl -n istio-system apply -f attribute_gen_service.yaml

    4. Create a local file system copy of the EnvoyFilter configuration, using the following command:

      1. $ kubectl -n istio-system get envoyfilter stats-filter-1.10 -o yaml > stats-filter-1.10.yaml

    5. Open stats-filter-1.10.yaml with a text editor and locate the name: istio.stats extension configuration. Update it to map response_code dimension in the requests_total standard metric to istio_responseClass attribute. The updated configuration file section should look like the following.

      1. name: istio.stats
      2. typed_config:
      3.   '@type': type.googleapis.com/udpa.type.v1.TypedStruct
      4.   type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
      5.   value:
      6.     config:
      7.       configuration:
      8.         "@type": type.googleapis.com/google.protobuf.StringValue
      9.         value: |
      10.           {
      11.             "metrics": [
      12.              {
      13.                "name": "requests_total",
      14.                "dimensions": {
      15.                  "response_code": "istio_responseClass"
      16.                }
      17.              }]
      18.           }

    6. Save stats-filter-1.10.yaml and then apply the configuration using the following command:

      1. $ kubectl -n istio-system apply -f stats-filter-1.10.yaml

    1. Generate metrics by sending traffic to your application.

    2. Visit Prometheus and look for the new or changed dimensions, for example 2xx. Alternatively, use the following command to verify that Istio generates the data for your new dimension:

      1. $ kubectl exec pod-name -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_

      In the output, locate the metric (e.g. istio_requests_total) and verify the presence of the new or changed dimension.

    If classification does not occur as expected, check the following potential causes and resolutions.

    Review the Envoy proxy logs for the pod that has the service on which you applied the configuration change. Check that there are no errors reported by the service in the Envoy proxy logs on the pod, (pod-name), where you configured classification by using the following command:

    Additionally, ensure that there are no Envoy proxy crashes by looking for signs of restarts in the output of the following command: