Experiment with the Kubeflow Pipelines API

    This tutorial demonstrates how to use the Kubeflow Pipelines API to build, run, and manage pipelines. This guide is recommended for users who would like to learn how to manage Kubeflow Pipelines using the REST API.

    This tutorial assumes that you have access to the service. If Kubeflow is not configured to use an identity provider, use port-forwarding to directly access the service.

    This tutorial assumes that the service is accessible on localhost.

    Building and running a pipeline

    Follow this guide to download, compile, and run the . To learn how to compile and run pipelines using the Kubeflow Pipelines SDK or a Jupyter notebook, follow the experimenting with Kubeflow Pipelines samples tutorial.

    1. PIPELINE_URL=https://raw.githubusercontent.com/kubeflow/pipelines/master/samples/core/sequential/sequential.py
    2. PIPELINE_FILE=${PIPELINE_URL##*/}
    3. PIPELINE_NAME=${PIPELINE_FILE%.*}
    5. wget -O ${PIPELINE_FILE} ${PIPELINE_URL}
    6. dsl-compile --py ${PIPELINE_FILE} --output ${PIPELINE_NAME}.tar.gz

    After running the commands above, you should get two files in your current directory: sequential.py and sequential.tar.gz. Run the following command to deploy the generated .tar.gz file as you would do using the , but this time using the REST API.

    If the operation was successful, you should see the pipeline in the central dashboard. You can also get the details using the PIPELINE_ID with the following API call.

    1. curl ${SVC}/apis/v1beta1/pipelines/${PIPELINE_ID} | jq

    Finally, use the PIPELINE_ID to trigger a run of your pipeline.

    1. RUN_ID=$((
    2. curl -H "Content-Type: application/json" -X POST ${SVC}/apis/v1beta1/runs \
    3. -d @- << EOF
    5.    "name":"${PIPELINE_NAME}_run",
    6.    "pipeline_spec":{
    7.    }
    8. }
    9. EOF
    10. ) | jq -r .run.id)

    Run the following command occasionally to see how the status of your run changes. After a while, the status of your pipeline should change to Succeeded.

    The response should be similar to the following one:

    1. {
    2.   "run": {
    3.     "id": "4ff0debd-d6d7-4681-8593-21ec002e6e0c",
    4.     "name": "sequential_run",
    5.     "pipeline_spec": {
    6.       "pipeline_id": "d30d28d7-0bfc-4f0c-8a57-6844a8ec9742",
    7.       "pipeline_name": "sequential.tar.gz",
    8.       "workflow_manifest": "{...}"
    9.     },
    11.       {
    12.           "type": "EXPERIMENT",
    13.           "id": "27af7eee-ce0a-44ba-a44d-07142abfc83c"
    14.         },
    15.         "name": "Default",
    16.         "relationship": "OWNER"
    17.       }
    18.     ],
    19.     "created_at": "2020-02-20T16:18:58Z",
    20.     "scheduled_at": "1970-01-01T00:00:00Z",
    21.     "finished_at": "1970-01-01T00:00:00Z",
    22.     "status": "Succeeded"
    23.   },
    24.   "pipeline_runtime": {
    25.     "workflow_manifest": "{...}"