Traffic Management Problems

    In the default access log format, Envoy response flags and Mixer policy status are located after the response code, if you are using a custom log format, make sure to include and %DYNAMIC_METADATA(istio.mixer:status)%.

    Refer to the for details of response flags.

    Common response flags are:

    • NR: No route configured, check your DestinationRule or VirtualService.
    • UO: Upstream overflow with circuit breaking, check your circuit breaker configuration in DestinationRule.
    • UF: Failed to connect to upstream, if you’re using Istio authentication, check for a mutual TLS configuration conflict.

    A request is rejected by Mixer if the response flag is UAEX and the Mixer policy status is not -.

    Common Mixer policy statuses are:

    • UNAVAILABLE: Envoy cannot connect to Mixer and the policy is configured to fail close.
    • UNAUTHENTICATED: The request is rejected by Mixer authentication.
    • PERMISSION_DENIED: The request is rejected by Mixer authorization.
    • RESOURCE_EXHAUSTED: The request is rejected by Mixer quota.
    • INTERNAL: The request is rejected due to Mixer internal error.

    Route rules don’t seem to affect traffic flow

    With the current Envoy sidecar implementation, up to 100 requests may be required for weighted version distribution to be observed.

    If route rules are working perfectly for the sample, but similar version routing rules have no effect on your own application, it may be that your Kubernetes services need to be changed slightly. Kubernetes services must adhere to certain restrictions in order to take advantage of Istio’s L7 routing features. Refer to the Requirements for Pods and Services for details.

    Another potential issue is that the route rules may simply be slow to take effect. The Istio implementation on Kubernetes utilizes an eventually consistent algorithm to ensure all Envoy sidecars have the correct configuration including all route rules. A configuration change will take some time to propagate to all the sidecars. With large deployments the propagation will take longer and there may be a lag time on the order of seconds.

    You should only see this error if you disabled during install.

    If requests to a service immediately start generating HTTP 503 errors after you applied a DestinationRule and the errors continue until you remove or revert the DestinationRule, then the DestinationRule is probably causing a TLS conflict for the service.

    For example, if you configure mutual TLS in the cluster globally, the DestinationRule must include the following trafficPolicy:

    1. trafficPolicy:
    2.   tls:
    3.     mode: ISTIO_MUTUAL

    Otherwise, the mode defaults to DISABLE causing client proxy sidecars to make plain HTTP requests instead of TLS encrypted requests. Thus, the requests conflict with the server proxy because the server proxy expects encrypted requests.

    Whenever you apply a DestinationRule, ensure the trafficPolicy TLS mode matches the global server configuration.

    Route rules have no effect on ingress gateway requests

    Let’s assume you are using an ingress Gateway and corresponding VirtualService to access an internal service. For example, your VirtualService looks something like this:

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: VirtualService
    3. metadata:
    4.   name: myapp
    5. spec:
    6.   hosts:
    7.   - "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello)
    8.   gateways:
    9.   - myapp-gateway
    10.   http:
    11.   - match:
    12.     - uri:
    13.         prefix: /hello
    14.     route:
    15.     - destination:
    16.         host: helloworld.default.svc.cluster.local
    17.   - match:
    18.     ...

    You also have a VirtualService which routes traffic for the helloworld service to a particular subset:

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: VirtualService
    3. metadata:
    4.   name: helloworld
    5. spec:
    6.   hosts:
    7.   - helloworld.default.svc.cluster.local
    8.   http:
    9.   - route:
    10.     - destination:
    11.         host: helloworld.default.svc.cluster.local
    12.         subset: v1

    In this situation you will notice that requests to the helloworld service via the ingress gateway will not be directed to subset v1 but instead will continue to use default round-robin routing.

    To control the traffic from the gateway, you need to also include the subset rule in the myapp VirtualService:

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: VirtualService
    3. metadata:
    4.   name: myapp
    5. spec:
    6.   hosts:
    7.   - "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello)
    8.   gateways:
    9.   - myapp-gateway
    10.   http:
    11.   - match:
    12.     - uri:
    13.         prefix: /hello
    14.     route:
    15.     - destination:
    16.         host: helloworld.default.svc.cluster.local
    17.         subset: v1
    18.   - match:
    19.     ...

    Alternatively, you can combine both VirtualServices into one unit if possible:

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: VirtualService
    3. metadata:
    4. spec:
    5.   hosts:
    6.   - myapp.com # cannot use "*" here since this is being combined with the mesh services
    7.   - helloworld.default.svc.cluster.local
    8.   gateways:
    9.   - mesh # applies internally as well as externally
    10.   - myapp-gateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         prefix: /hello
    15.       gateways:
    16.       - myapp-gateway #restricts this rule to apply only to ingress gateway
    17.     route:
    19.         host: helloworld.default.svc.cluster.local
    20.         subset: v1
    21.   - match:
    22.     - gateways:
    23.       - mesh # applies to all services inside the mesh
    24.     route:
    25.     - destination:
    26.         host: helloworld.default.svc.cluster.local
    27.         subset: v1

    Check your ulimit -a. Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with:

    Make sure to raise your ulimit. Example: ulimit -n 16384

    Envoy won’t connect to my HTTP/1.0 service

    Envoy requires HTTP/1.1 or HTTP/2 traffic for upstream services. For example, when using NGINX for serving traffic behind Envoy, you will need to set the directive in your NGINX configuration to be “1.1”, since the NGINX default is 1.0.

    Example configuration:

    1. upstream http_backend {
    2.     server 127.0.0.1:8080;
    4.     keepalive 16;
    5. }
    7. server {
    8.     ...
    10.     location /http/ {
    11.         proxy_pass http://http_backend;
    12.         proxy_http_version 1.1;
    13.         proxy_set_header Connection "";
    14.         ...
    15.     }
    16. }

    Many traffic management problems are caused by incorrect TLS configuration. The following sections describe some of the most common misconfigurations.

    If your application sends an HTTPS request to a service declared to be HTTP, the Envoy sidecar will attempt to parse the request as HTTP while forwarding the request, which will fail because the HTTP is unexpectedly encrypted.

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: ServiceEntry
    3. metadata:
    4.   name: httpbin
    5. spec:
    6.   hosts:
    7.   - httpbin.org
    8.   ports:
    9.   - number: 443
    10.     name: http
    11.     protocol: HTTP
    12.   resolution: DNS

    Although the above configuration may be correct if you are intentionally sending plaintext on port 443 (e.g., curl http://httpbin.org:443), generally port 443 is dedicated for HTTPS traffic.

    Sending an HTTPS request like curl https://httpbin.org, which defaults to port 443, will result in an error like curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number. The access logs may also show an error like 400 DPE.

    To fix this, you should change the port protocol to HTTPS:

    1. spec:
    2.   ports:
    3.   - number: 443
    4.     name: https
    5.     protocol: HTTPS

    There are two common TLS mismatches that can occur when binding a virtual service to a gateway.

    1. The gateway terminates TLS while the virtual service configures TLS routing.
    2. The gateway does TLS passthrough while the virtual service configures HTTP routing.

    Gateway with TLS termination

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: Gateway
    3. metadata:
    4.   name: gateway
    5.   namespace: istio-system
    6. spec:
    7.   selector:
    8.     istio: ingressgateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     hosts:
    15.       - "*"
    16.     tls:
    17.       mode: SIMPLE
    18.       credentialName: sds-credential
    19. apiVersion: networking.istio.io/v1beta1
    20. kind: VirtualService
    21. metadata:
    22.   name: httpbin
    23. spec:
    24.   hosts:
    25.   - "*.example.com"
    26.   gateways:
    27.   - istio-system/gateway
    28.   tls:
    29.   - match:
    30.     - sniHosts:
    31.       - "*.example.com"
    32.     route:
    33.     - destination:
    34.         host: httpbin.org

    In this example, the gateway is terminating TLS while the virtual service is using TLS based routing. The TLS route rules will have no effect since the TLS is already terminated when the route rules are evaluated.

    With this misconfiguration, you will end up getting 404 responses because the requests will be sent to HTTP routing but there are no HTTP routes configured. You can confirm this using the istioctl proxy-config routes command.

    To fix this problem, you should switch the virtual service to specify http routing, instead of tls:

    1. spec:
    2.   ...
    3.   http:
    4.   - match: ...

    Gateway with TLS passthrough

    In this configuration, the virtual service is attempting to match HTTP traffic against TLS traffic passed through the gateway. This will result in the virtual service configuration having no effect. You can observe that the HTTP route is not applied using the istioctl proxy-config listener and istioctl proxy-config route commands.

    1. spec:
    2.   tls:
    3.   - match:
    4.     - sniHosts: ["httpbin.example.com"]
    5.     route:
    6.     - destination:
    7.         host: httpbin.org

    Alternatively, you could terminate TLS, rather than passing it through, by switching the configuration in the gateway:

    1. spec:
    2.   ...
    3.     tls:
    4.       credentialName: sds-credential
    5.       mode: SIMPLE

    When configuring Istio to perform TLS origination, you need to make sure that the application sends plaintext requests to the sidecar, which will then originate the TLS.

    The following DestinationRule originates TLS for requests to the httpbin.org service, but the corresponding ServiceEntry defines the protocol as HTTPS on port 443.

    1. apiVersion: networking.istio.io/v1beta1
    2. kind: ServiceEntry
    3. metadata:
    4.   name: httpbin
    5. spec:
    6.   hosts:
    7.   - httpbin.org
    8.   ports:
    9.   - number: 443
    10.     name: https
    11.     protocol: HTTPS
    12.   resolution: DNS
    13. ---
    14. apiVersion: networking.istio.io/v1beta1
    15. kind: DestinationRule
    16. metadata:
    17.   name: originate-tls
    18. spec:
    19.   host: httpbin.org
    20.   trafficPolicy:
    21.     tls:
    22.       mode: SIMPLE

    With this configuration, the sidecar expects the application to send TLS traffic on port 443 (e.g., curl https://httpbin.org), but it will also perform TLS origination before forwarding requests. This will cause the requests to be double encrypted.

    For example, sending a request like curl https://httpbin.org will result in an error: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number.

    You can fix this example by changing the port protocol in the ServiceEntry to HTTP:

    1. spec:
    2.   hosts:
    3.   - httpbin.org
    4.   ports:
    5.   - number: 443
    6.     name: http
    7.     protocol: HTTP

    Note that with this configuration your application will need to send plaintext requests to port 433, like curl http://httpbin.org:443, because TLS origination does not change the port. However, starting in Istio 1.8, you can expose HTTP port 80 to the application (e.g., curl http://httpbin.org) and then redirect requests to targetPort 443 for the TLS origination:

    1. spec:
    2.   hosts:
    3.   - httpbin.org
    4.   ports:
    5.   - number: 80
    6.     name: http
    7.     protocol: HTTP
    8.     targetPort: 443

    Configuring more than one gateway using the same TLS certificate will cause browsers that leverage (i.e., most browsers) to produce 404 errors when accessing a second host after a connection to another host has already been established.

    For example, let’s say you have 2 hosts that share the same TLS certificate like this:

    • Wildcard certificate *.test.com installed in istio-ingressgateway
    • Gateway configuration gw1 with host service1.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • Gateway configuration gw2 with host service2.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • VirtualService configuration vs1 with host service1.test.com and gateway gw1
    • VirtualService configuration vs2 with host service2.test.com and gateway gw2

    Since both gateways are served by the same workload (i.e., selector istio: ingressgateway) requests to both services (service1.test.com and service2.test.com) will resolve to the same IP. If service1.test.com is accessed first, it will return the wildcard certificate (*.test.com) indicating that connections to service2.test.com can use the same certificate. Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to service2.test.com. Since the gateway (gw1) has no route for service2.test.com, it will then return a 404 (Not Found) response.

    You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2). Then, simply bind both VirtualServices to it like this:

    • Gateway configuration gw with host *.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • VirtualService configuration vs1 with host service1.test.com and gateway gw
    • VirtualService configuration vs2 with host service2.test.com and gateway gw

    An HTTPS Gateway that specifies the hosts field will perform an SNI match on incoming requests. For example, the following configuration would only allow requests that match *.example.com in the SNI:

    This may cause certain requests to fail.

    For example, if you do not have DNS set up and are instead directly setting the host header, such as curl 1.2.3.4 -H "Host: app.example.com", no SNI will be set, causing the request to fail. Instead, you can set up DNS or use the --resolve flag of curl. See the task for more information.

    Another common issue is load balancers in front of Istio. Most cloud load balancers will not forward the SNI, so if you are terminating TLS in your cloud load balancer you may need to do one of the following:

    • Disable SNI matching in the Gateway by setting the hosts field to

    A common symptom of this is for the load balancer health checks to succeed while real traffic fails.