Configuration Validation Problems

    The most commonly reported problems with configuration are YAML indentation and array notation (-) mistakes.

    Manually verify your configuration is correct, cross-referencing when necessary.

    Verify the istiod-istio-system validationwebhookconfiguration exists and is correct. The apiVersion, apiGroup, and resource of the invalid configuration should be listed in one of the two webhooks entries.

    If the validatingwebhookconfiguration doesn’t exist, verify the istio-validation configmap exists. Istio uses the data from this configmap to create and update the validatingwebhookconfiguration.

    1. $ kubectl -n istio-system get configmap istio-validation -o jsonpath='{.data}'
    2. map[config:apiVersion: admissionregistration.k8s.io/v1beta1
    3. kind: ValidatingWebhookConfiguration
    4. metadata:
    5.   name: istiod-istio-system
    6.   namespace: istio-system
    7.   labels:
    8.     app: istiod
    9.     release: istio
    10.     istio: istiod
    11. webhooks:
    12.   - name: validation.istio.io
    13.     clientConfig:
    14.       service:
    15.         name: istiod
    16.         namespace: istio-system
    18.       caBundle: ""
    19.     rules:
    20.       - operations:
    21.         - CREATE
    22.         - UPDATE
    23.         apiGroups:
    24.         - config.istio.io
    25.         - rbac.istio.io
    26.         - security.istio.io
    27.         - authentication.istio.io
    28.         - networking.istio.io
    29.         apiVersions:
    30.         - "*"
    31.         resources:
    32.         - "*"
    33.     failurePolicy: Fail
    34.     sideEffects: None]
    35.       (... snip ...)

    The validation configuration is fail-close. If configuration exists and is scoped properly, the webhook will be invoked. A missing caBundle, bad certificate, or network connectivity problem will produce an error message when the resource is created/updated. If you don’t see any error message and the webhook wasn’t invoked and the webhook configuration is valid, your cluster is misconfigured.

    x509: certificate signed by unknown authority related errors are typically caused by an empty caBundle in the webhook configuration. Verify that it is not empty (see verify webhook configuration). Istio consciously reconciles webhook configuration used the istio-validation configmap and root certificate.

    1. Verify the istiod pod(s) are running:

      1. $  kubectl -n istio-system get pod -lapp=istiod

    2. Check the pod logs for errors. Failing to patch the caBundle should print an error.

      1. $ kubectl get clusterrole istiod-istio-system -o yaml
      2. apiVersion: rbac.authorization.k8s.io/v1
      3. kind: ClusterRole
      4.   name: istiod-istio-system
      5. rules:
      6. - apiGroups:
      7.   - admissionregistration.k8s.io
      8.   resources:
      9.   - validatingwebhookconfigurations
      10.   verbs:
      11.   - '*'

      Istio needs validatingwebhookconfigurations write access to create and update the validatingwebhookconfiguration.

    Validation is fail-close. If the istiod pod is not ready, configuration cannot be created and updated. In such cases you’ll see an error about no endpoints available.

    Verify the istiod pod(s) are running and endpoints are ready.

    1. $  kubectl -n istio-system get pod -lapp=istiod
    2. NAME                            READY     STATUS    RESTARTS   AGE
    3. istiod-5dbbbdb746-d676g   1/1       Running   0          2d

    If the pods or endpoints aren’t ready, check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic.

    1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}'); do \
    2.     kubectl -n istio-system logs ${pod} \
    3. done
    1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o name); do \
    2.     kubectl -n istio-system describe ${pod} \