Monitor Cloud IAP Setup

    Cloud Identity-Aware Proxy (Cloud IAP) isthe recommended solution for accessing your Kubeflowdeployment from outside the cluster, when running Kubeflow on Google CloudPlatform (GCP).

    This document is a step-by-step guide to ensuring that your IAP-secured endpointis available, and to debugging problems that may cause the endpoint to beunavailable.

    When deploying Kubeflow using the or the command-line interface,you choose the authentication method you want to use. One of the options isCloud IAP. This document assumes that you have already deployed Kubeflow.

    Kubeflow uses the to provide an SSL certificate for the Kubeflow Ingress.

    Cloud IAP gives you the following benefits:

    • Users can log in in using their GCP accounts.
    • You benefit from Google’s security expertise to protect your sensitiveworkloads.
    • Examine theIngressand Google Cloud Build (GCB) load balancer to make sure it is available:

    There should be an annotation indicating that we are using managed certificate:

    2.   networking.gke.io/managed-certificates: gke-certificate

    Any problems with creating the load balancer are reported as Kubernetesevents in the results of the above describe command.

    • If the address isn’t set then there was a problem creating the loadbalancer.

    • The CREATE event indicates the load balancer was successfullycreated on the specified IP address.

    • Verify that a managed certificate resource is generated:
    • Wait for the load balancer to report the back ends as healthy:
    1.  kubectl describe -n istio-system ingress envoy-ingress
    3.  ...
    5.   kubernetes.io/ingress.global-static-ip-name:  kubeflow-ip
    6.   kubernetes.io/tls-acme:                       true
    7.   certmanager.k8s.io/issuer:                    letsencrypt-prod
    8.   ingress.kubernetes.io/backends:               {"k8s-be-31380--5e1566252944dfdb":"HEALTHY","k8s-be-32133--5e1566252944dfdb":"HEALTHY"}

    Both backends should be reported as healthy.It can take several minutes for the load balancer to consider the back endshealthy.

    The service with port 31380 is the one that handles Kubeflowtraffic. (31380 is the default port of the service istio-ingressgateway.)

    If the backend is unhealthy, check the pods in :

    • kubectl get pods -n istio-system
    • The istio-ingressgateway-XX pods should be running
    • Check the logs of pod backend-updater-0, iap-enabler-XX to see if there is any error
    • Follow the steps to check the load balancer and backend service on GCP.
      • Try accessing Cloud IAP at the fully qualified domain name in your webbrowser:

    If you get SSL errors when you log in, this typically means that your SSLcertificate is still propagating. Wait a few minutes and try again. SSLpropagation can take up to 10 minutes.

    If you do not see a login prompt and you get a 404 error, the configurationof Cloud IAP is not yet complete. Keep retrying for up to 10 minutes.

    • If you get an error Error: redirect_uri_mismatch after logging in, this means the list of OAuth authorized redirect URIs does not include your domain.
    1. The redirect URI in the request, https://mykubeflow.endpoints.myproject.cloud.goog/_gcp_gatekeeper/authenticate, does not match the ones authorized for the OAuth client.   
    2. To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/22222222222-7meeee7a9a76jvg54j0g2lv8lrsb4l8g.apps.googleusercontent.com?project=22222222222

    Follow the link in the error message to find the OAuth credential being usedand add the redirect URI listed in the error message to the list ofauthorized URIs. For more information, read the guide tosetting up OAuth for Cloud IAP.