Security Problems

    1. If isn’t set, make sure the JWT issuer is of url format and url + /.well-known/openid-configuration can be opened in browser; for example, if the JWT issuer is https://accounts.google.com, make sure https://accounts.google.com/.well-known/openid-configuration is a valid url and can be opened in a browser.

    2. If the JWT token is placed in the Authorization header in http requests, make sure the JWT token is valid (not expired, etc). The fields in a JWT token can be decoded by using online JWT parsing tools, e.g., jwt.io.

    3. Verify the Envoy proxy configuration of the target workload using istioctl proxy-config command.

      With the example policy above applied, use the following command to check the listener configuration on the inbound port 80. You should see envoy.filters.http.jwt_authn filter with settings matching the issuer and JWKS as specified in the policy.

      1. $ POD=$(kubectl get pod -l app=httpbin -n foo -o jsonpath={.items..metadata.name})
      2. $ istioctl proxy-config listener ${POD} -n foo --port 80 --type HTTP -o json
      3. <redacted>
      4.                             {
      5.                                 "name": "envoy.filters.http.jwt_authn",
      6.                                 "typedConfig": {
      7.                                     "@type": "type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication",
      8.                                     "providers": {
      9.                                         "origins-0": {
      10.                                             "issuer": "testing@secure.istio.io",
      11.                                             "localJwks": {
      12.                                                 "inlineString": "*redacted*"
      13.                                             },
      14.                                             "payloadInMetadata": "testing@secure.istio.io"
      15.                                         }
      16.                                     },
      17.                                     "rules": [
      18.                                         {
      19.                                             "match": {
      20.                                                 "prefix": "/"
      21.                                             },
      22.                                             "requires": {
      23.                                                 "requiresAny": {
      24.                                                     "requirements": [
      25.                                                         {
      26.                                                             "providerName": "origins-0"
      27.                                                         },
      28.                                                         {
      29.                                                             "allowMissing": {}
      30.                                                         }
      31.                                                     ]
      32.                                                 }
      33.                                             }
      34.                                         }
      35.                                     ]
      36.                                 }
      37.                             },
      38. <redacted>

    Authorization is too restrictive or permissive

    One common mistake is specifying multiple items unintentionally in the YAML. Take the following policy as an example:

    1. apiVersion: security.istio.io/v1beta1
    2. kind: AuthorizationPolicy
    3. metadata:
    4.   name: example
    5.   namespace: foo
    6. spec:
    7.   action: ALLOW
    8.   rules:
    9.   - to:
    10.     - operation:
    11.         paths:
    12.         - /foo
    13.   - from:
    14.     - source:
    15.         namespaces:
    16.         - foo

    You may expect the policy to allow requests if the path is /foo and the source namespace is foo. However, the policy actually allows requests if the path is /foo or the source namespace is foo, which is more permissive.

    In the YAML syntax, the - in front of the from: means it’s a new element in the list. This creates 2 rules in the policy instead of 1. In authorization policy, multiple rules have the semantics of OR.

    To fix the problem, just remove the extra - to make the policy have only 1 rule that allows requests if the path is /foo and the source namespace is foo, which is more restrictive.

    The authorization policy will be more restrictive because HTTP-only fields (e.g. host, path, headers, JWT, etc.) do not exist in the raw TCP connections.

    In the case of ALLOW policy, these fields are never matched. In the case of DENY and CUSTOM action, these fields are considered always matched. The final effect is a more restrictive policy that could cause unexpected denies.

    Check the Kubernetes service definition to verify that the port is named with the correct protocol properly. If you are using HTTP-only fields on the port, make sure the port name has the http- prefix.

    Check the workload selector and namespace to confirm it’s applied to the correct targets. You can determine the authorization policy in effect by running istioctl x authz check POD-NAME.POD-NAMESPACE.

    • If not specified, the policy defaults to use action ALLOW.

    • When a workload has multiple actions (, ALLOW and DENY) applied at the same time, all actions must be satisfied to allow a request. In other words, a request is denied if any of the action denies and is allowed only if all actions allow.

    • The AUDIT action does not enforce access control and will not deny the request at any cases.

    Istiod converts and distributes your authorization policies to the proxies. The following steps help you ensure Istiod is working as expected:

    1. Run the following command to open the Istiod ControlZ UI Page:

      1. $ istioctl dashboard controlz $(kubectl -n istio-system get pods -l app=istiod -o jsonpath='{.items[0].metadata.name}').istio-system

    2. Change the authorization Output Level to debug.

    4. Print the log of Istiod and search for authorization with the following command:

      You probably need to first delete and then re-apply your authorization policies so that the debug output is generated for these policies.

    5. Check the output and verify:

      • There are no errors.
      • There is a building v1beta1 policy message which indicates the filter was generated for the target workload.

    6. For example, you might see something similar to the following:

      1. 2020-03-05T23:43:21.621339Z   debug   authorization   found authorization allow policies for workload [app=ext-authz-server,pod-template-hash=5fd587cc9d,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=ext-authz-server,service.istio.io/canonical-revision=latest] in foo
      2. 2020-03-05T23:43:21.621348Z   debug   authorization   building filter for HTTP listener protocol
      3. 2020-03-05T23:43:21.621351Z   debug   authorization   building v1beta1 policy
      4. 2020-03-05T23:43:21.621399Z   debug   authorization   constructed internal model: &{Permissions:[{Services:[] Hosts:[] NotHosts:[] Paths:[] NotPaths:[] Methods:[] NotMethods:[] Ports:[] NotPorts:[] Constraints:[] AllowAll:true v1beta1:true}] Principals:[{Users:[] Names:[cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account] NotNames:[] Group: Groups:[] NotGroups:[] Namespaces:[] NotNamespaces:[] IPs:[] NotIPs:[] RequestPrincipals:[] NotRequestPrincipals:[] Properties:[] AllowAll:false v1beta1:true}]}
      5. 2020-03-05T23:43:21.621528Z   info    ads    LDS: PUSH for node:sleep-6bdb595bcb-vmchz.foo listeners:38
      6. 2020-03-05T23:43:21.621997Z   debug   authorization   generated policy ns[foo]-policy[ext-authz-server]-rule[0]: permissions:<and_rules:<rules:<any:true > > > principals:<and_ids:<ids:<or_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"source.principal" > value:<string_match:<exact:"cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account" > > > > > > > >
      7. 2020-03-05T23:43:21.622052Z   debug   authorization   added HTTP filter to filter chain 0
      8. 2020-03-05T23:43:21.623532Z   debug   authorization   found authorization allow policies for workload [app=ext-authz-server,pod-template-hash=5fd587cc9d,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=ext-authz-server,service.istio.io/canonical-revision=latest] in foo
      9. 2020-03-05T23:43:21.623543Z   debug   authorization   building filter for TCP listener protocol
      10. 2020-03-05T23:43:21.623546Z   debug   authorization   building v1beta1 policy
      11. 2020-03-05T23:43:21.623572Z   debug   authorization   constructed internal model: &{Permissions:[{Services:[] Hosts:[] NotHosts:[] Paths:[] NotPaths:[] Methods:[] NotMethods:[] Ports:[] NotPorts:[] Constraints:[] AllowAll:true v1beta1:true}] Principals:[{Users:[] Names:[cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account] NotNames:[] Group: Groups:[] NotGroups:[] Namespaces:[] NotNamespaces:[] IPs:[] NotIPs:[] RequestPrincipals:[] NotRequestPrincipals:[] Properties:[] AllowAll:false v1beta1:true}]}
      12. 2020-03-05T23:43:21.623625Z   debug   authorization   generated policy ns[foo]-policy[ext-authz-server]-rule[0]: permissions:<and_rules:<rules:<any:true > > > principals:<and_ids:<ids:<or_ids:<ids:<authenticated:<principal_name:<exact:"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account" > > > > > > >
      13. 2020-03-05T23:43:21.623645Z   debug   authorization   added TCP filter to filter chain 0
      14. 2020-03-05T23:43:21.623648Z   debug   authorization   added TCP filter to filter chain 1

      This shows that Istiod generated:

      • An HTTP filter config with policy ns[foo]-policy[ext-authz-server]-rule[0] for workload with labels app=ext-authz-server,....

      • A TCP filter config with policy ns[foo]-policy[ext-authz-server]-rule[0] for workload with labels app=ext-authz-server,....

    Ensure Istiod distributes policies to proxies correctly

    Pilot distributes the authorization policies to proxies. The following steps help you ensure Pilot is working as expected:

    The command used in this section assumes you have deployed Bookinfo application, otherwise you should replace "-l app=productpage" with your actual pod.

    1. Run the following command to get the proxy configuration dump for the productpage service:

      1. $ kubectl exec  $(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}') -c istio-proxy -- pilot-agent request GET config_dump

    2. Check the log and verify:

      • The log includes an envoy.filters.http.rbac filter to enforce the authorization policy on each incoming request.
      • Istio updates the filter accordingly after you update your authorization policy.

    3. The following output means the proxy of productpage has enabled the envoy.filters.http.rbac filter with rules that allows anyone to access it via GET method. The shadow_rules are not used and you can ignored them safely.

      1. {
      2.  "name": "envoy.filters.http.rbac",
      3.  "config": {
      4.   "rules": {
      5.    "policies": {
      6.     "productpage-viewer": {
      7.      "permissions": [
      8.       {
      9.        "and_rules": {
      10.         "rules": [
      11.          {
      12.           "or_rules": {
      13.            "rules": [
      14.             {
      15.              "header": {
      16.               "exact_match": "GET",
      17.               "name": ":method"
      18.              }
      19.             }
      20.            ]
      21.           }
      22.          }
      23.         ]
      24.        }
      25.       }
      26.      ],
      27.      "principals": [
      28.       {
      29.        "and_ids": {
      30.         "ids": [
      31.          {
      32.           "any": true
      33.          }
      34.         ]
      35.        }
      36.       }
      37.      ]
      38.     }
      39.    }
      40.   },
      41.   "shadow_rules": {
      42.    "policies": {}
      43.   }
      45. },

    Proxies eventually enforce the authorization policies. The following steps help you ensure the proxy is working as expected:

    The command used in this section assumes you have deployed . otherwise you should replace "-l app=productpage" with your actual pod.

    1. Verify you see the following output:

      1. active loggers:
      2.   ... ...
      3.   rbac: debug
      4.   ... ...

    2. Visit the productpage in your browser to generate some logs.

    3. Print the proxy logs with the following command:

      1. $ kubectl logs $(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}') -c istio-proxy

    4. Check the output and verify:

      • Your authorization policy expects the data extracted from the request.

    5. The following output means there is a GET request at path /productpage and the policy allows the request. The shadow denied has no effect and you can ignore it safely.

      1. ...
      2. [2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:79] checking request: remoteAddress: 10.60.0.139:51158, localAddress: 10.60.0.93:9080, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account, subjectPeerCertificate: O=, headers: ':authority', '35.238.0.62'
      3. ':path', '/productpage'
      4. ':method', 'GET'
      5. 'upgrade-insecure-requests', '1'
      6. 'user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
      7. 'dnt', '1'
      8. 'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'
      9. 'accept-encoding', 'gzip, deflate'
      10. 'accept-language', 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
      11. 'x-forwarded-for', '10.60.0.1'
      12. 'x-forwarded-proto', 'http'
      13. 'x-request-id', 'e23ea62d-b25d-91be-857c-80a058d746d4'
      14. 'x-b3-traceid', '5983108bf6d05603'
      15. 'x-b3-spanid', '5983108bf6d05603'
      16. 'x-b3-sampled', '1'
      17. 'x-istio-attributes', 'CikKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRINEgtwcm9kdWN0cGFnZQoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTc2NjY0Y2NmY2Ytd3hjcjQuaXN0aW8tc3lzdGVtCj4KE2Rlc3RpbmF0aW9uLnNlcnZpY2USJxIlcHJvZHVjdHBhZ2UuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbApDChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSJxIlcHJvZHVjdHBhZ2UuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbApBChdkZXN0aW5hdGlvbi5zZXJ2aWNlLnVpZBImEiRpc3RpbzovL2RlZmF1bHQvc2VydmljZXMvcHJvZHVjdHBhZ2U='
      18. 'content-length', '0'
      19. 'x-envoy-internal', 'true'
      20. 'sec-istio-authn-payload', 'CkVjbHVzdGVyLmxvY2FsL25zL2lzdGlvLXN5c3RlbS9zYS9pc3Rpby1pbmdyZXNzZ2F0ZXdheS1zZXJ2aWNlLWFjY291bnQSRWNsdXN0ZXIubG9jYWwvbnMvaXN0aW8tc3lzdGVtL3NhL2lzdGlvLWluZ3Jlc3NnYXRld2F5LXNlcnZpY2UtYWNjb3VudA=='
      21. , dynamicMetadata: filter_metadata {
      22.   key: "istio_authn"
      23.   value {
      24.     fields {
      25.       key: "request.auth.principal"
      26.       value {
      27.         string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      28.       }
      29.     }
      30.     fields {
      31.       key: "source.principal"
      32.       value {
      33.         string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      34.       }
      35.     }
      36.   }
      37. }
      39. [2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:88] shadow denied
      40. [2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:98] enforced allowed
      41. ...

    Keys and certificates errors

    If you suspect that some of the keys and/or certificates used by Istio aren’t correct, you can inspect the contents from any pod:

    By passing the -o json flag, you can pass the full certificate content to openssl to analyze its contents:

    1. $ istioctl proxy-config secret sleep-8f795f47d-4s4t7 -o json | jq '[.dynamicActiveSecrets[] | select(.name == "default")][0].secret.tlsCertificate.certificateChain.inlineBytes' -r | base64 -d | openssl x509 -noout -text
    2. Certificate:
    3.     Data:
    4.         Version: 3 (0x2)
    5.         Serial Number:
    6.             99:59:6b:a2:5a:f4:20:f4:03:d7:f0:bc:59:f5:d8:40
    7.     Signature Algorithm: sha256WithRSAEncryption
    8.         Issuer: O = k8s.cluster.local
    9.         Validity
    10.             Not Before: Jun  4 20:38:20 2018 GMT
    11.             Not After : Sep  2 20:38:20 2018 GMT
    12. ...
    13.         X509v3 extensions:
    14.             X509v3 Key Usage: critical
    15.                 Digital Signature, Key Encipherment
    16.             X509v3 Extended Key Usage:
    17.                 TLS Web Server Authentication, TLS Web Client Authentication
    18.             X509v3 Basic Constraints: critical
    19.                 CA:FALSE
    20.             X509v3 Subject Alternative Name:
    21.                 URI:spiffe://cluster.local/ns/my-ns/sa/my-sa

    Make sure the displayed certificate contains valid information. In particular, the Subject Alternative Name field should be URI:spiffe://cluster.local/ns/my-ns/sa/my-sa.

    If you suspect problems with mutual TLS, first ensure that , and second ensure that keys and certificates are being delivered to sidecars properly.

    If everything appears to be working so far, the next step is to verify that the right is applied and the right destination rules are in place.

    See also

    A tutorial to help customers migrate from the deprecated v1alpha1 security policy to the supported v1beta1 version.

    Istio in 2020 - Following the Trade Winds

    A vision statement and roadmap for Istio in 2020.

    A more secure way to manage secrets.

    DNS Certificate Management

    Provision and manage DNS certificates in Istio.

    Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

    Secure Webhook Management

    A more secure way to manage Istio webhooks.