Traffic Management Problems

    In the default access log format, Envoy response flags and Mixer policy status are located after the response code,if you are using a custom log format, make sure to include and %DYNAMIC_METADATA(istio.mixer:status)%.

    Refer to the Envoy response flagsfor details of response flags.

    Common response flags are:

    • NR: No route configured, check your DestinationRule or VirtualService.
    • UO: Upstream overflow with circuit breaking, check your circuit breaker configuration in DestinationRule.
    • UF: Failed to connect to upstream, if you’re using Istio authentication, check for a.

    A request is rejected by Mixer if the response flag is UAEX and the Mixer policy status is not -.

    Common Mixer policy statuses are:

    • UNAVAILABLE: Envoy cannot connect to Mixer and the policy is configured to fail close.
    • UNAUTHENTICATED: The request is rejected by Mixer authentication.
    • PERMISSION_DENIED: The request is rejected by Mixer authorization.
    • RESOURCE_EXHAUSTED: The request is rejected by Mixer quota.
    • INTERNAL: The request is rejected due to Mixer internal error.

    Route rules don’t seem to affect traffic flow

    With the current Envoy sidecar implementation, up to 100 requests may be required for weightedversion distribution to be observed.

    If route rules are working perfectly for the sample,but similar version routing rules have no effect on your own application, it may be thatyour Kubernetes services need to be changed slightly.Kubernetes services must adhere to certain restrictions in order to take advantage ofIstio’s L7 routing features.Refer to the Requirements for Pods and Servicesfor details.

    Another potential issue is that the route rules may simply be slow to take effect.The Istio implementation on Kubernetes utilizes an eventually consistentalgorithm to ensure all Envoy sidecars have the correct configurationincluding all route rules. A configuration change will take some timeto propagate to all the sidecars. With large deployments thepropagation will take longer and there may be a lag time on theorder of seconds.

    503 errors after setting destination rule

    If requests to a service immediately start generating HTTP 503 errors after you applied a DestinationRuleand the errors continue until you remove or revert the DestinationRule, then the DestinationRule is probablycausing a TLS conflict for the service.

    For example, if you configure mutual TLS in the cluster globally, the DestinationRule must include the following trafficPolicy:

    1. trafficPolicy:
    2.   tls:
    3.     mode: ISTIO_MUTUAL

    Otherwise, the mode defaults to DISABLE causing client proxy sidecars to make plain HTTP requestsinstead of TLS encrypted requests. Thus, the requests conflict with the server proxy because the server proxy expectsencrypted requests.

    1. $ istioctl authn tls-check istio-ingressgateway-db454d49b-lmtg8.istio-system httpbin.default.svc.cluster.local
    2. HOST:PORT                                  STATUS       SERVER     CLIENT     AUTHN POLICY     DESTINATION RULE
    3. httpbin.default.svc.cluster.local:8000     CONFLICT     mTLS       HTTP       default/         httpbin/default

    Whenever you apply a DestinationRule, ensure the trafficPolicy TLS mode matches the global server configuration.

    Let’s assume you are using an ingress Gateway and corresponding VirtualService to access an internal service.For example, your VirtualService looks something like this:

    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: VirtualService
    3. metadata:
    4.   name: myapp
    5. spec:
    6.   hosts:
    7.   - "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello)
    8.   gateways:
    9.   - myapp-gateway
    10.   http:
    11.   - match:
    12.     - uri:
    13.         prefix: /hello
    14.     route:
    15.     - destination:
    16.         host: helloworld.default.svc.cluster.local
    17.   - match:
    18.     ...

    You also have a VirtualService which routes traffic for the helloworld service to a particular subset:

    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: VirtualService
    3. metadata:
    4.   name: helloworld
    5. spec:
    6.   hosts:
    7.   - helloworld.default.svc.cluster.local
    8.   http:
    9.     - destination:
    10.         host: helloworld.default.svc.cluster.local
    11.         subset: v1

    In this situation you will notice that requests to the helloworld service via the ingress gateway willnot be directed to subset v1 but instead will continue to use default round-robin routing.

    The ingress requests are using the gateway host (e.g., myapp.com)which will activate the rules in the myapp VirtualService that routes to any endpoint of the helloworld service.Only internal requests with the host helloworld.default.svc.cluster.local will use thehelloworld VirtualService which directs traffic exclusively to subset v1.

    To control the traffic from the gateway, you need to also include the subset rule in the myapp VirtualService:

    Alternatively, you can combine both VirtualServices into one unit if possible:

    2. kind: VirtualService
    3. metadata:
    4.   name: myapp
    5. spec:
    6.   hosts:
    7.   - myapp.com # cannot use "*" here since this is being combined with the mesh services
    8.   - helloworld.default.svc.cluster.local
    9.   gateways:
    10.   - mesh # applies internally as well as externally
    11.   - myapp-gateway
    12.   http:
    13.   - match:
    14.     - uri:
    15.         prefix: /hello
    16.       gateways:
    17.       - myapp-gateway #restricts this rule to apply only to ingress gateway
    18.     route:
    19.     - destination:
    20.         host: helloworld.default.svc.cluster.local
    21.         subset: v1
    22.   - match:
    23.     - gateways:
    24.       - mesh # applies to all services inside the mesh
    25.     route:
    26.     - destination:
    27.         host: helloworld.default.svc.cluster.local
    28.         subset: v1

    Headless TCP services losing connection

    If istio-citadel is deployed, Envoy is restarted every 45 days to refresh certificates.This causes the disconnection of TCP streams or long-running connections between services.

    You should build resilience into your application for this type ofdisconnect, but if you still want to prevent the disconnects fromhappening, you will need to disable mutual TLS and the istio-citadel deployment.

    First, edit your istio configuration to disable mutual TLS:

    1. $ kubectl edit configmap -n istio-system istio
    2. $ kubectl delete pods -n istio-system -l istio=pilot

    Next, scale down the istio-citadel deployment to disable Envoy restarts:

    1. $ kubectl scale --replicas=0 deploy/istio-citadel -n istio-system

    Envoy is crashing under load

    Check your ulimit -a. Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with:

    1. [2017-05-17 03:00:52.735][14236][critical][assert] assert failure: fd_ != -1: external/envoy/source/common/network/connection_impl.cc:58

    Make sure to raise your ulimit. Example: ulimit -n 16384

    Envoy requires HTTP/1.1 or HTTP/2 traffic for upstream services. For example, when using NGINX for serving traffic behind Envoy, youwill need to set the directive in your NGINX configuration to be “1.1”, since the NGINX default is 1.0.

    Example configuration:

    404 errors occur when multiple gateways configured with same TLS certificate

    Configuring more than one gateway using the same TLS certificate will cause browsersthat leverage (i.e., most browsers) to produce 404 errors when accessing a second host after aconnection to another host has already been established.

    For example, let’s say you have 2 hosts that share the same TLS certificate like this:

    • Wildcard certificate *.test.com installed in istio-ingressgateway
    • Gateway configuration gw1 with host service1.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • Gateway configuration gw2 with host service2.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • VirtualService configuration vs1 with host service1.test.com and gateway gw1
    • VirtualService configuration vs2 with host service2.test.com and gateway gw2

    Since both gateways are served by the same workload (i.e., selector istio: ingressgateway) requests to both services(service1.test.com and service2.test.com) will resolve to the same IP. If service1.test.com is accessed first, itwill return the wildcard certificate (*.test.com) indicating that connections to service2.test.com can use the same certificate.Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to service2.test.com.Since the gateway (gw1) has no route for service2.test.com, it will then return a 404 (Not Found) response.

    You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2).Then, simply bind both VirtualServices to it like this:

    • Gateway configuration gw with host , selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
    • VirtualService configuration vs1 with host service1.test.com and gateway gw
    • VirtualService configuration vs2 with host service2.test.com and gateway gw

    Port conflict when configuring multiple TLS hosts in a gateway

    If you apply a Gateway configuration that has the same selector labels as anotherexisting Gateway, then if they both expose the same HTTPS port you must ensure that they haveunique port names. Otherwise, the configuration will be applied without an immediate error indicationbut it will be ignored in the runtime gateway configuration. For example:

    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: Gateway
    3. metadata:
    4.   name: mygateway
    5. spec:
    6.   selector:
    7.   servers:
    8.   - port:
    9.       number: 443
    10.       name: https
    11.       protocol: HTTPS
    12.     tls:
    13.       mode: SIMPLE
    14.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    15.       privateKey: /etc/istio/ingressgateway-certs/tls.key
    16.     hosts:
    17.     - "myhost.com"
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: Gateway
    21. metadata:
    22.   name: mygateway2
    23. spec:
    24.   selector:
    25.     istio: ingressgateway # use istio default ingress gateway
    26.   servers:
    27.   - port:
    28.       number: 443
    29.       name: https
    30.       protocol: HTTPS
    31.     tls:
    32.       mode: SIMPLE
    33.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    34.       privateKey: /etc/istio/ingressgateway-certs/tls.key
    35.     hosts:
    36.     - "myhost2.com"

    With this configuration, requests to the second host, myhost2.com, will fail becauseboth gateway ports have name: https.A curl request, for example, will produce an error message something like this:

    1. curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhost2.com:443

    You can confirm that this has happened by checking Pilot’s logs for a message similar to the following:

    1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=pilot -n istio-system -o jsonpath={.items..metadata.name}) -c discovery | grep "non unique port"
    2. 2018-09-14T19:02:31.916960Z info    model   skipping server on gateway mygateway2 port https.443.HTTPS: non unique port name for HTTPS port
    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: Gateway
    3. metadata:
    4.   name: mygateway2
    5. spec:
    6.   selector:
    7.     istio: ingressgateway # use istio default ingress gateway
    8.   servers:
    9.   - port:
    10.       number: 443
    11.       name: https2
    12.       protocol: HTTPS
    13.     tls:
    14.       mode: SIMPLE
    15.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    16.       privateKey: /etc/istio/ingressgateway-certs/tls.key
    17.     - "myhost2.com"