Pipelines Standalone Deployment

As an alternative to as awhole with many components including pipelines, you also have a choice to deployonly Kubeflow Pipelines. Follow the instructions below to deployKubeflow Pipelines standalone using the supplied kustomizemanifests.

Knowledge about Kubernetes, and kustomize will help you understand thisdocument better and be able to customize your deployment based on your needs.

Installation outside GCP is also available

This guide currently describes how to install Kubeflow Pipelines standalone on Google Cloud Platform (GCP). You can also install Kubeflow Pipelines on otherplatforms. This guide needs updating. Seeissue #1253.

These are common one time setups you need for all the instructions below:

Follow the .

You need kubectl version 1.14 or later, for native support of kustomize.

Configure kubectl to talk to your cluster

See the Google Kubernetes Engine (GKE) guide to .

Deploying Kubeflow Pipelines standalone to an existing cluster

• Deploy the latest version of Kubeflow Pipelines:

Note: The above approach is introduced in Kubeflow Pipelines version 0.2.0. For older versions please run the following instead:

kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION

• Get the URL for the Kubeflow Pipelines UI :

kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com

• Prepare a Kubernetes cluster:

See the GKE guide to for Google Cloud Platform (GCP).

Use the following gcloud command to create a cluster that can run all pipeline samples:

# The following parameters can be customized based on your needs.
CLUSTER_NAME="kubeflow-pipelines-standalone"
MACHINE_TYPE="n1-standard-2" # A machine with 2 CPUs and 7.50GB memory
SCOPES="storage-rw,cloud-platform" # These scopes are needed for running some pipeline samples.
gcloud container clusters create $CLUSTER_NAME \
    --zone $ZONE \
    --machine-type $MACHINE_TYPE \
    --scopes $SCOPES \
    --num-nodes 2 \
    --max-nodes 5 \
    --min-nodes 2 \

Reference:

• You can find available zones in https://cloud.google.com/compute/docs/regions-zones/#available.
• Get gcloud CLI tool at .
• Read gcloud container clusters create command documentation at https://cloud.google.com/sdk/gcloud/reference/container/clusters/create.
  • Configure kubectl to talk to your newly created cluster. Refer to .

• Deploy the latest version of Kubeflow Pipelines standalone to your cluster:

export PIPELINE_VERSION=0.2.0
kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/base/crds?ref=$PIPELINE_VERSION
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION

Kubeflow Pipelines applications take a while (~3 minutes) to start.

Note: The above approach is introduced in Kubeflow Pipelines version 0.2.0. For older versions please run the following instead:

• Get public URL of Pipelines UI and use it to access Kubeflow Pipelines:

kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com

Upgrade

• Configure kubectl to talk to your cluster. Refer to .
• Upgrade to a version of Kubeflow Pipelines standalone you choose:

export PIPELINE_VERSION=<version-you-want-to-upgrade-to>
kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION

Check Kubeflow Pipelines github repo for available releases.

Customization can be done through kustomize .

Note - The instruction below assume you installed kubectl v1.14.0 or later, which has native support of kustomize.To get latest kubectl, visit here

For the following instructions, first clone ,and use it as working directory.

Deploy on GCP with CloudSQL and Google Cloud Storage

This is recommended for production environments. See for more details.

To deploy Kubeflow Pipelines standalone in namespace FOO:

• Edit dev/kustomization.yamlor namespace section to FOO.

kubectl apply -k manifests/kustomize/env/dev
# Or the following if using GCP Cloud SQL + Google Cloud Storage
# kubectl apply -k manifests/kustomize/env/gcp

Disable the public endpoint

By default, the deployment installs an that exposes a public URL. If you want to skip installing it,

• Comment out the proxy component in the kustomization.yaml.
• Then run:

kubectl apply -k manifests/kustomize/env/dev
# Or the following if using GCP Cloud SQL + Google Cloud Storage
# kubectl apply -k manifests/kustomize/env/gcp

and open .

Uninstall

You can uninstall Kubeflow Pipelines by:

• Configure kubectl to talk to your cluster. Refer to .

• Uninstall Pipelines:

export PIPELINE_VERSION=0.2.0

Or if you deployed through kustomize:

kubectl delete -k manifests/kustomize/env/dev
# Or the following if using GCP Cloud SQL + Google Cloud Storage
# kubectl delete -k manifests/kustomize/env/gcp

Maintain a repo for your manifests

Save the following to a source controlled repo.

File kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Edit the following to change the deployment to your custom namespace.
namespace: kubeflow
# You can add other customizations here using kustomize.
# Edit ref in the following link to deploy a different version of Kubeflow Pipelines.
bases:
- github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=0.2.0

Deploy: kubectl apply -k $YOUR_REPO

Upgrade:

• (Recommended) Back up your data storages for KFP.
• Edit ref=0.2.0 to a version you want to upgrade to.

Check [Kubeflow Pipelines github repo](https://github.com/kubeflow/pipelines/releases) for available releases.

• Deploy: kubectl apply -k $YOUR_REPO.

Uninstall: .

Further reading

Troubleshooting

Permission error installing Kubeflow Pipelines standalone to a cluster