Before you can use HPA in your Kubernetes cluster, you must fulfill some requirements.

Be sure that your Kubernetes cluster services are running with these flags at minimum:

  • kube-api:
  • kubelet: read-only-port at 10255

  • kube-controller: Optional, just needed if distinct values than default are required.

    • horizontal-pod-autoscaler-downscale-delay: "5m0s"
    • horizontal-pod-autoscaler-upscale-delay: "3m0s"
    • horizontal-pod-autoscaler-sync-period: "30s"

For an RKE Kubernetes cluster definition, add this snippet in the services section. To add this snippet using the Rancher v2.0 UI, open the Clusters view and select ⋮ > Edit for the cluster in which you want to use HPA. Then, from Cluster Options, click Edit as YAML. Add the following snippet to the services section:

Once the Kubernetes cluster is configured and deployed, you can deploy metrics services.

To create HPA resources based on resource metrics such as CPU and memory use, you need to deploy the metrics-server package in the kube-system namespace of your Kubernetes cluster. This deployment allows HPA to consume the metrics.k8s.io API.

Prerequisite: You must be running kubectl 1.8 or later.

  1. Connect to your Kubernetes cluster using kubectl.

    1. # git clone https://github.com/kubernetes-incubator/metrics-server

  3. Install the metrics-server package.

    1. # kubectl create -f metrics-server/deploy/1.8+/

  4. Check that metrics-server is running properly. Check the service pod and logs in the kube-system namespace.

    1. Check the service pod for a status of running. Enter the following command:

      1. # kubectl get pods -n kube-system

      Then check for the status of running.

    2. Check the service logs for service availability. Enter the following command:

      1. # kubectl -n kube-system logs metrics-server-6fbfb84cdd-t2fk9

      Then review the log to confirm that the metrics-server package is running.

      Metrics Server Log Output

      1. I0723 08:09:56.193136       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
      3. I0723 08:09:56.194480       1 configs.go:61] Using Kubernetes client with master "https://10.43.0.1:443" and version
      4. I0723 08:09:56.198612       1 heapster.go:128] Starting with Metric Sink
      5. I0723 08:09:56.780114       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
      6. I0723 08:09:57.391518       1 heapster.go:101] Starting Heapster API server...
      7. [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
      8. [restful] 2018/07/23 08:09:57 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
      9. I0723 08:09:57.394080       1 serve.go:85] Serving securely on 0.0.0.0:443

  5. Check that the metrics api is accessible from kubectl.

    • If you are accessing the cluster through Rancher, enter your Server URL in the kubectl config in the following format: https://<RANCHER_URL>/k8s/clusters/<CLUSTER_ID>. Add the suffix /k8s/clusters/<CLUSTER_ID> to API path.

      1. # kubectl get --raw /k8s/clusters/<CLUSTER_ID>/apis/metrics.k8s.io/v1beta1

    • If you are accessing the cluster directly, enter your Server URL in the kubectl config in the following format: https://<K8s_URL>:6443.

      1. # kubectl get --raw /apis/metrics.k8s.io/v1beta1

      If the API is working correctly, you should receive output similar to the output below.

      1. {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]}

By default, HPA reads resource and custom metrics with the user system:anonymous. Assign system:anonymous to view-resource-metrics and view-custom-metrics in the ClusterRole and ClusterRoleBindings manifests. These roles are used to access metrics.

To do it, follow these steps:

  1. Configure kubectl to connect to your cluster.

  2. Copy the ClusterRole and ClusterRoleBinding manifest for the type of metrics you’re using for your HPA.

    Resource Metrics: ApiGroups resource.metrics.k8s.io

    1.     apiVersion: rbac.authorization.k8s.io/v1
    2.     kind: ClusterRole
    3.     metadata:
    5.     rules:
    6.     - apiGroups:
    7.         - metrics.k8s.io
    8.         - pods
    9.         - nodes
    10.       verbs:
    11.         - get
    12.         - list
    13.         - watch
    14.     ---
    15.     apiVersion: rbac.authorization.k8s.io/v1
    16.     kind: ClusterRoleBinding
    17.     metadata:
    18.       name: view-resource-metrics
    19.     roleRef:
    20.       apiGroup: rbac.authorization.k8s.io
    21.       kind: ClusterRole
    22.       name: view-resource-metrics
    23.     subjects:
    24.       - apiGroup: rbac.authorization.k8s.io
    25.         kind: User
    26.         name: system:anonymous

    Custom Metrics: ApiGroups custom.metrics.k8s.io

  3. Create them in your cluster using one of the follow commands, depending on the metrics you’re using.

    1. # kubectl create -f <RESOURCE_METRICS_MANIFEST>