Connecting to Kubeflow Pipelines on Google Cloud using the SDK

    This guide describes how to connect to your Kubeflow Pipelines cluster on Google Cloud using the Kubeflow Pipelines SDK.

    Kubeflow Pipelines includes an API service named . The ml-pipeline-ui API service is deployed in the same Kubernetes namespace you deployed Kubeflow Pipelines in.

    The Kubeflow Pipelines SDK can send REST API requests to this API service, but the SDK needs to know the hostname to connect to the API service.

    If the hostname can be accessed without authentication, it’s very simple to connect to it. For example, you can use kubectl port-forward to access it via localhost:

    When deploying Kubeflow Pipelines on Google Cloud, a public endpoint for this API service is auto-configured for you, but this public endpoint has security checks to protect your cluster from unauthorized access.

    The following sections introduce how to authenticate your SDK requests to connect to Kubeflow Pipelines via the public endpoint.

    Kubeflow Pipelines standalone deployments also show up in . They have the name “pipeline” by default, but you can customize the name by overriding the parameter in params.env when .

    A full Kubeflow deployment on Google Cloud uses an Identity-Aware Proxy (IAP) to manage access to the public Kubeflow endpoint. The steps below let you connect to Kubeflow Pipelines in a full Kubeflow deployment with authentication through IAP.

    1. Find out your IAP OAuth 2.0 client ID.

      You or your cluster admin followed to deploy your full Kubeflow deployment on GCP. You need the OAuth client ID created in that step.

      You can browse all of your existing OAuth client IDs in the Credentials page of Google Cloud Console.

    2. Create another SDK OAuth Client ID for authenticating Kubeflow Pipelines SDK users. Follow . Take a note of the client ID and client secret. This client ID and secret can be shared among all SDK users, because a separate login step is still needed below.

    3. When you init the SDK client for the first time, you will be asked to log in. The Kubeflow Pipelines SDK stores obtained credentials in $HOME/.config/kfp/credentials.json. You do not need to log in again unless you manually delete the credentials file.

    4. After login, you can use the client.