Monitor Cloud IAP Setup

    Cloud Identity-Aware Proxy (Cloud IAP) isthe recommended solution for accessing your Kubeflowdeployment from outside the cluster, when running Kubeflow on Google CloudPlatform (GCP).

    This document is a step-by-step guide to ensuring that your IAP-secured endpointis available, and to debugging problems that may cause the endpoint to beunavailable.

    When deploying Kubeflow using the or the command-line interface,you choose the authentication method you want to use. One of the options isCloud IAP. This document assumes that you have already deployed Kubeflow.

    Kubeflow uses the service to providean SSL certificate for the Kubeflow UI.

    Cloud IAP gives you the following benefits:

    • Users can log in in using their GCP accounts.
    • You benefit from Google’s security expertise to protect your sensitiveworkloads.

    Follow these instructions to monitor your Cloud IAP setup and troubleshoot anyproblems:

    • Examine theIngressand Google Cloud Build (GCB) load balancer to make sure it is available:

    Any problems with creating the load balancer are reported as Kubernetes events in the results of the above command.

    • If the address isn’t set then there was a problem creating the loadbalancer.

    • The most common error is running out of GCP quota. To fix this problem,you must either increase the quota for the relevant resource on your GCPproject or delete some existing resources.

    • Verify that a signed SSL certificate was generated from:
    1.   kubectl -n istio-system get certificate envoy-ingress-tls  -o yaml
    3.   apiVersion: certmanager.k8s.io/v1alpha1
    4.   kind: Certificate
    5.   metadata:
    6.     creationTimestamp: 2019-04-02T22:49:43Z
    7.     generation: 1
    8.     labels:
    9.       kustomize.component: iap-ingress
    10.     name: envoy-ingress-tls
    11.     namespace: istio-system
    12.     selfLink: /apis/certmanager.k8s.io/v1alpha1/namespaces/kubeflow/certificates/envoy-ingress-tls
    13.     uid: 9b137b29-5599-11e9-a223-42010a8e020c
    14.   spec:
    15.     acme:
    16.       config:
    17.       - domains:
    18.         - mykubeflow.endpoints.myproject.cloud.goog
    20.           ingress: envoy-ingress
    21.     commonName: kf-vmaster-n01.endpoints.kubeflow-ci-deployment.cloud.goog
    22.     dnsNames:
    23.     - mykubeflow.endpoints.myproject.cloud.goog
    24.     issuerRef:
    25.       kind: ClusterIssuer
    26.       name: letsencrypt-prod
    27.     secretName: envoy-ingress-tls
    28.   status:
    29.     acme:
    30.       order:
    31.         url: https://acme-v02.api.letsencrypt.org/acme/order/54483154/382580193
    32.     conditions:
    33.       message: Certificate issued successfully
    34.       reason: CertIssued
    35.       status: "True"
    36.       type: Ready
    37.     - lastTransitionTime: null
    38.       message: Order validated
    40.       status: "False"
    41.       type: ValidateFailed

    It can take around 10 minutes to provision a certificate after thecreation of the load balancer.

    The most recent condition should be Certificate issued successfully.

    The most common error is running out of Let’s Encryptquota.Let’s Encrypt enforces a quota of 5 duplicate certificates per week.

    The easiest fix to quota issues is to pick a different hostname byrecreating and redeploying Kubeflow with a differentname.

    For example if you originally ran the following kfctl init command:

    Then rerun kfctl init with a different name that you haven’t usedbefore:

    1. kfctl init myapp-unique --project=myproject --config=myconfig -V
    • Wait for the load balancer to report the back ends as healthy:

    Both backends should be reported as healthy.It can take several minutes for the load balancer to consider the back endshealthy.

    If the backend is unhealthy, check the pods in istio-system:

    • kubectl get pods -n istio-system
    • The istio-ingressgateway-XX pods should be running
    • Check the logs of backend-updater-0, ingress-bootstrap-XX, iap-enabler-XX to see if there is any error
      • Now that the certificate exists, the Ingress resource should report that itis serving on HTTPS:
    1. kubectl -n istio-system get ingress
    2. NAME            HOSTS                                                        ADDRESS          PORTS     AGE
    3. envoy-ingress   mykubeflow.endpoints.myproject.cloud.goog   35.244.132.159   80, 443   1d

    If you don’t see port 443, look at the Ingress events usingkubectl describe to see if there are any errors.

    • Try accessing Cloud IAP at the fully qualified domain name in your webbrowser:

    If you get SSL errors when you log in, this typically means that your SSLcertificate is still propagating. Wait a few minutes and try again. SSLpropagation can take up to 10 minutes.

    If you do not see a login prompt and you get a 404 error, the configurationof Cloud IAP is not yet complete. Keep retrying for up to 10 minutes.

    • If you get an error Error: redirect_uri_mismatch after logging in, thismeans the list of OAuth authorized redirect URIs does not include your domain.

    The full error message looks like the following example and includes therelevant links:

    1. The redirect URI in the request, https://mykubeflow.endpoints.myproject.cloud.goog/_gcp_gatekeeper/authenticate, does not match the ones authorized for the OAuth client.

    Follow the link in the error message to find the OAuth credential being usedand add the redirect URI listed in the error message to the list ofauthorized URIs. For more information, read the guide to.

    Kubeflow runs an agent in your cluster to renew the Let’s Encrypt certificateautomatically. You don’t need to take any action.For more information, see the Let’s Encryptdocumentation.

    For questions and support about the certificate, visit.