Galley Configuration Problems

    Verify the validationwebhookconfiguration exists andis correct. The apiVersion, apiGroup, and resource of theinvalid configuration should be listed in one of the two webhooksentries.

    If the validatingwebhookconfiguration doesn’t exist, verify theistio-galley-configuration configmap exists. istio-galley usesthe data from this configmap to create and update thevalidatingwebhookconfiguration.

    1. $ kubectl -n istio-system get configmap istio-galley-configuration -o jsonpath='{.data}'
    2. map[validatingwebhookconfiguration.yaml:apiVersion: admissionregistration.k8s.io/v1beta1
    3. kind: ValidatingWebhookConfiguration
    4. metadata:
    5.   name: istio-galley
    6.   namespace: istio-system
    7.   labels:
    8.     app: istio-galley
    9.     chart: galley-1.0.0
    10.     release: istio
    11.     heritage: Tiller
    12. webhooks:
    13.   - name: pilot.validation.istio.io
    14.     clientConfig:
    15.       service:
    16.         name: istio-galley
    17.         namespace: istio-system
    18.         path: "/admitpilot"
    19.       caBundle: ""
    20.       - operations:
    21.       (... snip ...)

    The istio-galley validation configuration is fail-close. Ifconfiguration exists and is scoped properly, the webhook will beinvoked. A missing caBundle, bad certificate, or network connectivityproblem will produce an error message when the resource iscreated/updated. If you don’t see any error message and the webhookwasn’t invoked and the webhook configuration is valid, your cluster ismisconfigured.

    x509: certificate signed by unknown authority related errors aretypically caused by an empty caBundle in the webhookconfiguration. Verify that it is not empty (see verify webhookconfiguration). Theistio-galley deployment consciously reconciles webhook configurationused the istio-galley-configuration configmap and root certificatemounted from istio.istio-galley-service-account secret in theistio-system namespace.

    • Verify the istio-galley pod(s) are running:
    1. $  kubectl -n istio-system get pod -listio=galley
    2. NAME                            READY     STATUS    RESTARTS   AGE
    3. istio-galley-5dbbbdb746-d676g   1/1       Running   0          2d
    • Verify you’re using Istio version >= 1.0.0. Older version of Galleydid not properly re-patch the caBundle. This typically happenedwhen the istio.yaml was re-applied, overwriting a previouslypatched caBundle.
    • Check the Galley pod logs for errors. Failing to patch thecaBundle should print an error.
    1. $ for pod in $(kubectl -n istio-system get pod -listio=galley -o jsonpath='{.items[*].metadata.name}'); do \
    2.     kubectl -n istio-system logs ${pod} \
    3. done
    • If the patching failed, verify the RBAC configuration for Galley:
    1. $ kubectl get clusterrole istio-galley-istio-system -o yaml
    2. apiVersion: rbac.authorization.k8s.io/v1
    3. kind: ClusterRole
    4. metadata:
    5.   labels:
    6.     app: istio-galley
    7.   name: istio-galley-istio-system
    8. rules:
    9. - apiGroups:
    10.   - admissionregistration.k8s.io
    12.   verbs:
    13.   - '*'
    14. - apiGroups:
    15.   - config.istio.io
    16.   resources:
    17.   - '*'
    18.   verbs:
    19.   - get
    20.   - list
    21.   - watch
    22. - apiGroups:
    23.   - '*'
    24.   resourceNames:
    25.   - istio-galley
    26.   resources:
    27.   - deployments
    28.   verbs:
    29.   - get

    Validation is fail-close. If the istio-galley pod is not ready,configuration cannot be created and updated. In such cases you’ll seean error about no endpoints available.

    Verify the istio-galley pod(s) are running and endpoints are ready.

    1. $ kubectl -n istio-system get endpoints istio-galley
    2. NAME           ENDPOINTS                          AGE
    3. istio-galley   10.48.6.108:15014,10.48.6.108:443   3d
    1. $ for pod in $(kubectl -n istio-system get pod -listio=galley -o jsonpath='{.items[*].metadata.name}'); do \
    2.     kubectl -n istio-system logs ${pod} \