Authenticating Kubeflow to GCP

    This page describes in-cluster and local authentication for Kubeflow GCP deployments.

    Starting from Kubeflow v0.6, you consume Kubeflow from custom namespaces (that is, namespaces other than ). The kubeflow namespace is only for running Kubeflow system components. Individual jobs and model deployments run in separate namespaces.

    Starting in v0.7, Kubeflow uses the new GKE feature: . This is the recommended way to access GCP APIs from your GKE cluster. You can configure a Kubernetes service account (KSA) to act as a GCP service account (GSA).

    If you deployed Kubeflow following the GCP instructions, then the profiler controller automatically binds the “default-editor” service account for every profile namespace to a default GCP service account created during kubeflow deployment. The Kubeflow deployment process also creates a default profile for the cluster admin.

    For more info about profiles see the Multi-user isolation page.

    Here is an example profile spec:

    You can verify that there is a KSA called default-editor and that it has an annotation of the corresponding GSA:

    1. kubectl -n ${PROFILE_NAME} describe serviceaccount default-editor
    3. ...
    4. Annotations: iam.gke.io/gcp-service-account: ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com
    5. ...

    You can double check that GSA is also properly set up:

    1. gcloud --project=${PROJECT} iam service-accounts get-iam-policy ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com

    When a pod uses KSA default-editor, it can access GCP APIs with the role granted to the GSA.

    Provisioning custom Google service accounts in namespaces: When creating a profile, you can specify a custom GCP service account for the namespace to control which GCP resources are accessible.

    Prerequisite: you must have permission to edit your GCP project’s IAM policy and to create a profile custom resource (CR) in your Kubeflow cluster.

    1. if you don’t already have a GCP service account you want to use, create a new one. For example: user1-gcp@<project-id>.iam.gserviceaccount.com:
    1. gcloud iam service-accounts create user1-gcp@<project-id>.iam.gserviceaccount.com
    1. You can bind roles to the GCP service account to allow access to the desired GCP resources. For example to run BigQuery job, you can grant access like so:
    1. of service account user1-gcp@<project-id>.iam.gserviceaccount.com to cluster account <cluster-name>-admin@<project-id>.iam.gserviceaccount.com:
    1. gcloud iam service-accounts add-iam-policy-binding \
    2.       user1-gcp@<project-id>.iam.gserviceaccount.com \
    3.       --member='serviceAccount:<cluster-name>-admin@<project-id>.iam.gserviceaccount.com' --role='roles/owner'
    1. Manually create a profile for user1 and specify the GCP service account to bind in plugins field:
    1. apiVersion: kubeflow.org/v1beta1
    2. kind: Profile
    4.   name: profileName   # replace with the name of the profile (the user's namespace name)
    5. spec:
    6.   owner:
    7.     name: user1@email.com   # replace with the email of the user
    8.   plugins:
    9.   - kind: WorkloadIdentity
    10.     spec:
    11.       gcpServiceAccount: user1-gcp@project-id.iam.gserviceaccount.com

    Note: The profile controller currently doesn’t perform any access control checks to see whether the user creating the profile should be able to use the GCP service account. As a result, any user who can create a profile can get access to any service account for which the admin controller has owner permissions. We will improve this in subsequent releases.

    Starting from Kubeflow v1.1, Kubeflow Pipelines supports multi-user isolation. Therefore, pipeline runs are executed in user namespaces also using the default-editor KSA.

    Additionally, the Kubeflow Pipelines UI, visualization, and TensorBoard server instances are deployed in your user namespace using the default-editor KSA. Therefore, to , they can fetch artifacts in Google Cloud Storage using permissions of the same GSA you configured for this namespace.

    For more details, refer to Authenticating Pipelines to GCP.

    Use the to interact with Google Cloud Platform (GCP) on the command line. You can use the gcloud command to set up Google Kubernetes Engine (GKE) clusters, and interact with other Google services.

    Logging in

    You have two options for authenticating the gcloud command:

    • You can use a user account to authenticate using a Google account (typically Gmail). You can register a user account using gcloud auth login, which brings up a browser window to start the familiar Google authentication flow.

    • You can create a service account within your GCP project. You can then associated with the account, and run the gcloud auth activate-service-account command to authenticate your gcloud session.

    You can find more information in the .

    Listing active accounts

    You can run the following command to verify you are authenticating with the expected account:

    1. gcloud auth list

    In the output of the command, an asterisk denotes your active account.

    Viewing IAM roles

    Permissions are handled in GCP using IAM Roles. These roles define which resources your account can read or write to. Provided you have the , you can check which roles were assigned to your account using the following gcloud command:

    You can view and modify roles through the GCP IAM console.

    The is used for interacting with a Kubernetes cluster through the command line.

    Connecting to a cluster using a GCP account

    If you set up your Kubernetes cluster using GKE, you can authenticate with the cluster using a GCP account. The following commands fetch the credentials for your cluster and save them to your local :

    1. CLUSTER_NAME=your-gke-cluster
    2. ZONE=your-gcp-zone
    4. gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE

    You can find more information in the GCP docs.

    Changing active clusters

    If you work with multiple Kubernetes clusters, you may have multiple contexts saved in your local kubeconfig file. You can view the clusters you have saved by run the following command:

    1. kubectl config get-contexts

    You can view which cluster is currently being controlled by kubectl with the following command:

    1. CONTEXT_NAME=your-new-context
    3. kubectl config set-context $CONTEXT_NAME

    You can find more information in the .

    Checking RBAC permissions

    Like GKE IAM, Kubernetes permissions are typically handled with a “role-based authorization control” (RBAC) system. Each Kubernetes service account has a set of authorized roles associated with it. If your account doesn’t have the right roles assigned to it, certain tasks fail.

    You can check if an account has the proper permissions to run a command by building a query structured as kubectl auth can-i [VERB] [RESOURCE] --namespace [NAMESPACE]. For example, the following command verifies that your account has permissions to create deployments in the kubeflow namespace:

    You can find more information in the .

    Adding RBAC permissions

    If you find you are missing a permission you need, you can grant the missing roles to your service account using Kubernetes resources.

    • Roles describe the permissions you want to assign. For example, verbs: ["create"], resources:["deployments"]

    By default, Roles and RoleBindings apply only to resources in a specific namespace, but there are also ClusterRoles and ClusterRoleBindings that can grant access to resources cluster-wide

    You can find more information in the .