MXNet Training

Alpha

This Kubeflow component has alpha status with limited support. See the . The Kubeflow team is interested in your feedback about the usability of the feature.

This guide walks you through using MXNet with Kubeflow.

If you haven’t already done so please follow the to deploy Kubeflow.

A version of MXNet support was introduced with Kubeflow 0.2.0. You must be using a version of Kubeflow newer than 0.2.0.

Verify that MXNet support is included in your Kubeflow deployment

Check that the MXNet custom resource is installed

  1. NAME                                           AGE
  2. ...
  3. mxjobs.kubeflow.org                            4d
  4. ...

If it is not included you can add it as follows

Alternatively, you can deploy the operator with default settings without using kustomize by running the following from the repo:

  1. git clone https://github.com/kubeflow/mxnet-operator.git
  2. cd mxnet-operator
  3. kubectl create -f manifests/crd-v1beta1.yaml 
  4. kubectl create -f manifests/rbac.yaml 
  5. kubectl create -f manifests/deployment.yaml

You create a training job by defining a MXJob with MXTrain mode and then creating it with

Creating a TVM tuning job (AutoTVM)

TVM is a end to end deep learning compiler stack, you can easily run AutoTVM with mxnet-operator.You can create a auto tuning job by define a type of MXTune job and then creating it with

  1. kubectl create -f examples/v1beta1/tune/mx_job_tune_gpu.yaml

Before you use the auto-tuning example, there is some preparatory work need to be finished in advance. To let TVM tune your network, you should create a docker image which has TVM module. Then, you need a auto-tuning script to specify which network will be tuned and set the auto-tuning parameters, For more details, please see . Finally, you need a startup script to start the auto-tuning program. In fact, mxnet-operator will set all the parameters as environment variables and the startup script need to reed these variable and then transmit them to auto-tuning script. We provide an example under examples/v1beta1/tune/, tuning result will be saved in a log file like resnet-18.log in the example we gave. You can refer it for details.

Here is sample output for an example job

  1. apiVersion: kubeflow.org/v1beta1
  2. kind: MXJob
  3. metadata:
  4.   creationTimestamp: 2019-03-19T09:24:27Z
  5.   generation: 1
  6.   name: mxnet-job
  7.   namespace: default
  8.   resourceVersion: "3681685"
  9.   selfLink: /apis/kubeflow.org/v1beta1/namespaces/default/mxjobs/mxnet-job
  10.   uid: cb11013b-4a28-11e9-b7f4-704d7bb59f71
  11. spec:
  12.   cleanPodPolicy: All
  13.   jobMode: MXTrain
  14.   mxReplicaSpecs:
  15.     Scheduler:
  16.       replicas: 1
  17.       restartPolicy: Never
  18.       template:
  19.         metadata:
  20.           creationTimestamp: null
  21.           containers:
  22.           - image: mxjob/mxnet:gpu
  23.             name: mxnet
  25.             - containerPort: 9091
  26.               name: mxjob-port
  27.             resources: {}
  28.     Server:
  29.       replicas: 1
  30.       restartPolicy: Never
  31.       template:
  32.         metadata:
  33.           creationTimestamp: null
  34.         spec:
  35.           containers:
  36.           - image: mxjob/mxnet:gpu
  37.             name: mxnet
  38.             ports:
  39.             - containerPort: 9091
  40.               name: mxjob-port
  41.             resources: {}
  42.     Worker:
  43.       replicas: 1
  44.       restartPolicy: Never
  45.       template:
  46.         metadata:
  47.           creationTimestamp: null
  48.         spec:
  49.           containers:
  50.           - args:
  51.             - /incubator-mxnet/example/image-classification/train_mnist.py
  52.             - --num-epochs
  53.             - "10"
  54.             - --num-layers
  55.             - "2"
  56.             - --kv-store
  57.             - dist_device_sync
  59.             command:
  60.             - python
  61.             image: mxjob/mxnet:gpu
  62.             name: mxnet
  63.             ports:
  64.             - containerPort: 9091
  65.               name: mxjob-port
  66.             resources:
  67.               limits:
  68.                 nvidia.com/gpu: "1"
  69. status:
  70.   completionTime: 2019-03-19T09:25:11Z
  71.   conditions:
  72.   - lastTransitionTime: 2019-03-19T09:24:27Z
  73.     lastUpdateTime: 2019-03-19T09:24:27Z
  74.     message: MXJob mxnet-job is created.
  75.     reason: MXJobCreated
  76.     status: "True"
  77.     type: Created
  78.   - lastTransitionTime: 2019-03-19T09:24:27Z
  79.     lastUpdateTime: 2019-03-19T09:24:29Z
  80.     message: MXJob mxnet-job is running.
  81.     reason: MXJobRunning
  82.     status: "False"
  83.     type: Running
  84.   - lastTransitionTime: 2019-03-19T09:24:27Z
  85.     lastUpdateTime: 2019-03-19T09:25:11Z
  86.     message: MXJob mxnet-job is successfully completed.
  87.     reason: MXJobSucceeded
  88.     status: "True"
  89.     type: Succeeded
  90.   mxReplicaStatuses:
  91.     Scheduler: {}
  92.     Server: {}
  94.   startTime: 2019-03-19T09:24:29Z

Feedback

Was this page helpful?

Glad to hear it! Please .

Sorry to hear that. Please tell us how we can improve.

Last modified 04.02.2020: