PyTorch Training

    This Kubeflow component has stable status. See the Kubeflow versioning policies.

    This guide walks you through using PyTorch with Kubeflow.

    If you haven’t already done so please follow the to deploy Kubeflow.

    An alpha version of PyTorch support was introduced with Kubeflow 0.2.0. You must be using a version of Kubeflow between 0.2.0 and 0.3.5 to use this version.

    Check that the PyTorch custom resource is installed

    The output should include 

    1. NAME                                           AGE
    2. ...
    3. pytorchjobs.kubeflow.org                       4d
    4. ...

    If it is not included you can add it as follows

    1. export KF_DIR=<your Kubeflow installation directory>
    2. cd ${KF_DIR}/kustomize
    3. kubectl apply -f pytorch-job-crds.yaml
    4. kubectl apply -f pytorch-operator.yaml

    You can create PyTorch Job by defining a PyTorchJob config file. See the manifests for the distributed MNIST example. You may change the config file based on your requirements.

    1. kubectl create -f pytorch_job_mnist.yaml

    You should now be able to see the created pods matching the specified number of replicas.

    1. kubectl get pods -l pytorch_job_name=pytorch-tcp-dist-mnist

    Training should run for about 10 epochs and takes 5-10 minutes on a cpu cluster. Logs can be inspected to see its training progress.

    1. kubectl get -o yaml pytorchjobs pytorch-tcp-dist-mnist

    See the status section to monitor the job status. Here is sample output when the job is successfully completed.

    1. apiVersion: kubeflow.org/v1
    2. kind: PyTorchJob
    3. metadata:
    4.   clusterName: ""
    5.   creationTimestamp: 2018-12-16T21:39:09Z
    6.   generation: 1
    7.   name: pytorch-tcp-dist-mnist
    8.   namespace: default
    9.   resourceVersion: "15532"
    10.   selfLink: /apis/kubeflow.org/v1/namespaces/default/pytorchjobs/pytorch-tcp-dist-mnist
    11.   uid: 059391e8-017b-11e9-bf13-06afd8f55a5c
    12.   cleanPodPolicy: None
    13.   pytorchReplicaSpecs:
    15.       replicas: 1
    16.       restartPolicy: OnFailure
    17.       template:
    18.         metadata:
    19.           creationTimestamp: null
    20.         spec:
    21.           containers:
    22.           - image: gcr.io/kubeflow-ci/pytorch-dist-mnist_test:1.0
    23.             name: pytorch
    24.             ports:
    25.             - containerPort: 23456
    26.               name: pytorchjob-port
    27.             resources: {}
    28.     Worker:
    29.       replicas: 3
    30.       restartPolicy: OnFailure
    31.       template:
    32.         metadata:
    33.           creationTimestamp: null
    34.         spec:
    35.           containers:
    36.           - image: gcr.io/kubeflow-ci/pytorch-dist-mnist_test:1.0
    37.             name: pytorch
    38.             - containerPort: 23456
    40.             resources: {}
    41. status:
    42.   completionTime: 2018-12-16T21:43:27Z
    43.   conditions:
    44.   - lastTransitionTime: 2018-12-16T21:39:09Z
    45.     lastUpdateTime: 2018-12-16T21:39:09Z
    46.     message: PyTorchJob pytorch-tcp-dist-mnist is created.
    47.     reason: PyTorchJobCreated
    48.     status: "True"
    49.     type: Created
    50.   - lastTransitionTime: 2018-12-16T21:39:09Z
    51.     lastUpdateTime: 2018-12-16T21:40:45Z
    52.     message: PyTorchJob pytorch-tcp-dist-mnist is running.
    53.     reason: PyTorchJobRunning
    54.     status: "False"
    55.     type: Running
    56.   - lastTransitionTime: 2018-12-16T21:39:09Z
    57.     lastUpdateTime: 2018-12-16T21:43:27Z
    58.     message: PyTorchJob pytorch-tcp-dist-mnist is successfully completed.
    59.     reason: PyTorchJobSucceeded
    60.     status: "True"
    61.     type: Succeeded
    62.   replicaStatuses:
    63.     Master: {}