Automatically scaling pods based on custom metrics

The custom metrics autoscaler uses the Kubernetes-based Event Driven Autoscaler (KEDA) and is built on top of the OKD horizontal pod autoscaler (HPA).

The Custom Metrics Autoscaler Operator scales your pods up and down based on custom, external metrics from specific applications. Your other applications continue to use other scaling methods. You configure triggers, also known as scalers, which are the source of events and metrics that the custom metrics autoscaler uses to determine how to scale. The custom metrics autoscaler uses a metrics API to convert the external metrics to a form that OKD can use. The custom metrics autoscaler creates a horizontal pod autoscaler (HPA) that performs the actual scaling. The custom metrics autoscaler currently supports only the Prometheus trigger, which can use the installed OKD monitoring or an external Prometheus server as the metrics source.

To use the custom metrics autoscaler, you create a or ScaledJob object, which defines the scaling metadata. You specify the deployment or job to scale, the source of the metrics to scale on (trigger), and other parameters such as the minimum and maximum replica counts allowed.

You can verify that the autoscaling has taken place by reviewing the number of pods in your custom resource or by reviewing the Custom Metrics Autoscaler Operator logs for messages similar to the following:

Successfully updated ScaleTarget

Installing the custom metrics autoscaler

You can use the OKD web console to install the Custom Metrics Autoscaler Operator.

The installation creates five CRDs:

• ClusterTriggerAuthentication

• KedaController

• ScaledJob

• ScaledObject

• TriggerAuthentication

Prerequisites

• Ensure that you have downloaded the as shown in Obtaining the installation program in the installation documentation for your platform.

  If you have the pull secret, add the redhat-operators catalog to the OperatorHub custom resource (CR) as shown in Configuring OKD to use Red Hat Operators.

• If you use the community KEDA:

  • Uninstall the community KEDA. You cannot run both KEDA and the custom metrics autoscaler on the same OKD cluster.

  • Remove the KEDA 1.x custom resource definitions by running the following commands:

    $ oc delete crd scaledobjects.keda.k8s.io

    $ oc delete crd triggerauthentications.keda.k8s.io

Procedure

1. In the OKD web console, click Operators → OperatorHub.

2. Choose Custom Metrics Autoscaler from the list of available Operators, and click Install.

3. On the Install Operator page, ensure that the All namespaces on the cluster (default) option is selected for Installation Mode. This installs the Operator in all namespaces.

4. Ensure that the openshift-keda namespace is selected for Installed Namespace. OKD creates the namespace, if not present in your cluster.

5. Click Install.

6. Verify the installation by listing the Custom Metrics Autoscaler Operator components:

   1. Navigate to Workloads → Pods.

   2. Select the openshift-keda project from the drop-down menu and verify that the custom-metrics-autoscaler-operator-* pod is running.

   3. Navigate to Workloads → Deployments to verify that the custom-metrics-autoscaler-operator deployment is running.

7. Optional: Verify the installation in the OpenShift CLI using the following commands:

   $ oc get all -n openshift-keda

   The output appears similar to the following:

   Example output

   NAME                                                      READY   STATUS    RESTARTS   AGE
   pod/custom-metrics-autoscaler-operator-5fd8d9ffd8-xt4xp   1/1     Running   0          18m
   NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/custom-metrics-autoscaler-operator   1/1     1            1           18m
   NAME                                                            DESIRED   CURRENT   READY   AGE
   replicaset.apps/custom-metrics-autoscaler-operator-5fd8d9ffd8   1         1         1       18m

8. Install the KedaController custom resource, which creates the required CRDs:

   1. In the OKD web console, click Operators → Installed Operators.

   2. Click Custom Metrics Autoscaler.

   3. On the Operator Details page, click the KedaController tab.

   4. On the KedaController tab, click Create KedaController and edit the file.

      kind: KedaController
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: keda
        namespace: openshift-keda
      spec:
        watchNamespace: '' (1)
        operator:
          logLevel: info (2)
          logEncoder: console (3)
        metricsServer:
          logLevel: '0' (4)
        serviceAccount: {}

      1	Specifies the namespaces that the custom autoscaler should watch. Enter names in a comma-separated list. Omit or set empty to watch all namespaces. The default is empty.
      2	Specifies the level of verbosity for the Custom Metrics Autoscaler Operator log messages. The allowed values are debug, info, error. The default is info.
      3	Specifies the logging format for the Custom Metrics Autoscaler Operator log messages. The allowed values are console or json. The default is console.
      4	Specifies the logging level for the Custom Metrics Autoscaler Metrics Server. The allowed values are 0 for info and 4 or debug. The default is 0.

   5. Click Create to create the KEDAController.

Triggers, also known as scalers, provide the metrics that the Custom Metrics Autoscaler Operator uses to scale your pods.

You use a ScaledObject or ScaledJob custom resource to configure triggers for specific objects, as described in the sections that follow.

Scale applications based on Prometheus metrics. See Additional resources for information on the configurations required to use the OKD monitoring as a source for metrics.

Example scaled object with a Prometheus target

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prom-scaledobject
  namespace: my-namespace
spec:
 ...
  triggers:
  - type: prometheus (1)
    metadata:
      serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 (2)
      namespace: kedatest (3)
      metricName: http_requests_total (4)
      threshold: '5' (5)
      query: sum(rate(http_requests_total{job="test-app"}[1m])) (6)
      authModes: "basic" (7)

Additional resources

• Configuring the custom metrics autoscaler to use OKD monitoring

Understanding custom metrics autoscaler trigger authentications

A trigger authentication allows you to include authentication information in a scaled object or a scaled job that can be used by the associated containers. You can use trigger authentications to pass OKD secrets, platform-native pod authentication mechanisms, environment variables, and so on.

You define a TriggerAuthentication object in the same namespace as the object that you want to scale. That trigger authentication can be used only by objects in that namespace.

Alternatively, to share credentials between objects in multiple namespaces, you can create a ClusterTriggerAuthentication object that can be used across all namespaces.

Trigger authentications and cluster trigger authentication use the same configuration. However, a cluster trigger authentication requires an additional kind parameter in the authentication reference of the scaled object.

Example trigger authentication with a secret

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: secret-triggerauthentication
  namespace: my-namespace (1)
spec:
  secretTargetRef: (2)
  - parameter: user-name (3)
    name: my-secret (4)
    key: USER_NAME (5)
  - parameter: password
    name: my-secret
    key: PASSWORD

Example cluster trigger authentication with a secret

kind: ClusterTriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata: (1)
  name: secret-cluster-triggerauthentication
spec:
  secretTargetRef: (2)
  - parameter: user-name (3)
    name: secret-name (4)
    key: user-name (5)
  - parameter: password
    name: secret-name
    key: user-name

Example trigger authentication with a token

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: token-triggerauthentication
  namespace: my-namespace (1)
spec:
  secretTargetRef: (2)
  - parameter: bearerToken (3)
    name: my-token-2vzfq (4)
    key: token (5)
  - parameter: ca
    name: my-token-2vzfq
    key: ca.crt

Example trigger authentication with an environment variable

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
  name: env-var-triggerauthentication
  namespace: my-namespace (1)
spec:
  env: (2)
  - parameter: access_key (3)
    name: ACCESS_KEY (4)
    containerName: my-container (5)

Example trigger authentication with pod authentication providers

kind: TriggerAuthentication
apiVersion: keda.sh/v1alpha1
metadata:
spec:
  podIdentity: (2)
    provider: aws-eks (3)

Additional resources

• For information on OKD secrets, see Providing sensitive data to pods.

You use trigger authentications and cluster trigger authentications by using a custom resource to create the authentication, then add a reference to a scaled object or scaled job.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

• If you are using a secret, the Secret object must exist, for example:

  Example secret

  apiVersion: v1
  kind: Secret
  metadata:
    name: my-secret
  data:
    user-name: <base64_username>
    password: <base64_password>

1. Create the TriggerAuthentication or ClusterTriggerAuthentication object.

   1. Create a YAML file that defines the object:

      Example trigger authentication with a secret

   2. Create the TriggerAuthentication object:

      $ oc create -f <file-name>.yaml

2. Create or edit a ScaledObject YAML file:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: scaledobject
     namespace: my-namespace
   spec:
     scaleTargetRef:
       name: example-deployment
     maxReplicaCount: 100
     minReplicaCount: 0
     pollingInterval: 30
     triggers:
     - authenticationRef:
       type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest # replace <NAMESPACE>
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "basic"
       - authenticationRef: (1)
           name: prom-triggerauthentication
         metadata:
           name: prom-triggerauthentication
         type: object
       - authenticationRef: (2)
           name: prom-cluster-triggerauthentication
           kind: ClusterTriggerAuthentication
         metadata:
           name: prom-cluster-triggerauthentication
         type: object

   1	Optional: Specify a trigger authentication.
   2	Optional: Specify a cluster trigger authentication. You must include the kind: ClusterTriggerAuthentication parameter.

   It is not necessary to specify both a namespace trigger authentication and a cluster trigger authentication.

3. Create the object. For example:

   $ oc apply -f <file-name>

You can use the installed OKD monitoring as a source for the metrics used by the custom metrics autoscaler. However, there are some additional configurations you must perform.

You must perform the following tasks, as described in this section:

• Create a service account to get a token.

• Create a role.

• Add that role to the service account.

• Reference the token in the trigger authentication object used by Prometheus.

Prerequisites

• OKD monitoring must be installed.

• Monitoring of user-defined workloads must be enabled in OKD monitoring, as described in the Creating a user-defined workload monitoring config map section.

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Change to the project with the object you want to scale:

   $ oc project my-project

2. Use the following command to create a service account, if your cluster does not have one:

   $ oc create serviceaccount <service_account>

   where:

   <service_account>

   Specifies the name of the service account.

3. Use the following command to locate the token assigned to the service account:

   $ oc describe serviceaccount <service_account>

   where:

   <service_account>

   Specifies the name of the service account.

   Example output

   Name:                thanos
   Namespace:           my-project
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-nnwgj
   Mountable secrets:   thanos-dockercfg-nnwgj
   Tokens:              thanos-token-9g4n5 (1)
   Events:              <none>

   1	Use this token in the trigger authentication.

4. Create a trigger authentication with the service account token:

   1. Create a YAML file similar to the following:

      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
      spec:
        secretTargetRef: (1)
        - parameter: bearerToken (2)
          name: thanos-token-9g4n5 (3)
          key: token (4)
        - parameter: ca
          name: thanos-token-9g4n5
          key: ca.crt

      1	Specifies that this object uses a secret for authorization.
      2	Specifies the authentication parameter to supply by using the token.
      3	Specifies the name of the token to use.
      4	Specifies the key in the token to use with the specified parameter.

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

5. Create a role for reading Thanos metrics:

   1. Create a YAML file with the following parameters:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

6. Create a role binding for reading Thanos metrics:

   1. Create a YAML file similar to the following:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: thanos-metrics-reader (1)
        namespace: my-project (2)
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: thanos-metrics-reader
      subjects:
      - kind: ServiceAccount
        name: thanos (3)
        namespace: my-project (4)

      1	Specifies the name of the role you created.
      2	Specifies the namespace of the object you want to scale.
      3	Specifies the name of the service account to bind to the role.
      4	Specifies the namespace of the object you want to scale.

   2. Create the CR object:

      $ oc create -f <file-name>.yaml

You can now deploy a scaled object or scaled job to enable autoscaling for your application, as described in the following sections. To use OKD monitoring as the source, in the trigger, or scaler, specify the prometheus type and use https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 as the serverAddress.

Additional resources

Understanding how to add custom metrics autoscalers

To add a custom metrics autoscaler, create a ScaledObject custom resource for a deployment, stateful set, or custom resource. Create a ScaledJob custom resource for a job.

You can create only one scaled object or scaled job for each workload that you want to scale. Also, you cannot use a scaled object or scaled job and the horizontal pod autoscaler (HPA) on the same workload.

You can create a custom metrics autoscaler for a workload created by a Deployment, StatefulSet, or custom resource` object.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Create a YAML file similar to the following:

   Example scaled object

   1	Optional: Specifies the API version of the target resource. The default is apps/v1.
   2	Specifies the name of the object that you want to scale.
   3	Specifies the as Deployment, StatefulSet or CustomResource.
   4	Optional: Specifies the name of the container in the target resource, from which the custom autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].
   5	Optional. Specifies the period in seconds to wait after the last trigger is reported before scaling the deployment back to 0 if the minReplicaCount is set to 0. The default is 300.
   6	Optional: Specifies the maximum number of replicas when scaling up. The default is 100.
   7	Optional: Specifies the minimum number of replicas when scaling down.
   8	Optional: Specifies the interval in seconds to check each trigger on. The default is 30.
   9	Optional: Specifies whether to scale back the target resource to the original replicas count after the scaled object is deleted. The default is false, which keeps the replica count as it is when the scaled object is deleted.
   10	Optional: Specifies a scaling policy to use to control the rate to scale pods up or down. For more information, see the link in the “Additional resources” section that follows.
   11	Specifies the trigger to use as the basis for scaling, as described in the “Understanding the custom metrics autoscaler triggers” section. This example uses OKD monitoring.
   12	Optional: Specifies a trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.
   13	Optional: Specifies a cluster trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.

   It is not necessary to specify both a namespace trigger authentication and a cluster trigger authentication.

2. Create the custom metrics autoscaler:

   $ oc create -f <file-name>.yaml

Verification

• View the command output to verify that the custom metrics autoscaler was created:

  $ oc get scaledobject <scaled_object_name>

  Example output

  NAME            SCALETARGETKIND      SCALETARGETNAME        MIN   MAX   TRIGGERS     AUTHENTICATION               READY   ACTIVE   FALLBACK   AGE
  scaledobject    apps/v1.Deployment   example-deployment     0     50    prometheus   prom-triggerauthentication   True    True     True       17s

  Note the following fields in the output:

• TRIGGERS: Indicates the trigger, or scaler, that is being used.

• AUTHENTICATION: Indicates the name of any trigger authentication being used.

• READY: Indicates whether the scaled object is ready to start scaling:

  • If True, the scaled object is ready.

  • If False, the scaled object is not ready because of a problem in one or more of the objects you created.

• ACTIVE: Indicates whether scaling is taking place:

  • If True, scaling is taking place.

  • If False, scaling is not taking place because there are no metrics or there is a problem in one or more of the objects you created.

• • If False, the custom metrics autoscaler is getting metrics.

  • If True, the custom metrics autoscaler is getting metrics because there are no metrics or there is a problem in one or more of the objects you created.

Additional resources

• Understanding the custom metrics autoscaler triggers

You can create a custom metrics autoscaler for any Job object.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Create a YAML file similar to the following:

   kind: ScaledJob
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: scaledjob
     namespace: my-namespace
   spec:
     failedJobsHistoryLimit: 5
     jobTargetRef:
       activeDeadlineSeconds: 600 (1)
       backoffLimit: 6 (2)
       parallelism: 1 (3)
       completions: 1 (4)
       template:  (5)
         metadata:
           name: pi
         spec:
           containers:
           - name: pi
             image: perl
             command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
     maxReplicaCount: 100 (6)
     pollingInterval: 30 (7)
     successfulJobsHistoryLimit: 5 (8)
     failedJobsHistoryLimit: 5 (9)
     envSourceContainerName: (10)
     rolloutStrategy: gradual (11)
     scalingStrategy: (12)
       strategy: "custom"
       customScalingQueueLengthDeduction: 1
       customScalingRunningJobPercentage: "0.5"
       pendingPodConditions:
         - "Ready"
         - "PodScheduled"
         - "AnyOtherCustomPodCondition"
       multipleScalersCalculation : "max"
     triggers:
     - type: prometheus (13)
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "bearer"
     - authenticationRef: (14)
         name: prom-triggerauthentication
       metadata:
         name: prom-triggerauthentication
        type: object
     - authenticationRef: (15)
         name: prom-cluster-triggerauthentication
       metadata:
         name: prom-cluster-triggerauthentication
       type: object

   1	Specifies the maximum duration the job can run.
   2	Specifies the number of retries for a job. The default is 6.
   3	Optional: Specifies how many pod replicas a job should run in parallel; defaults to 1. For non-parallel jobs, leave unset. When unset, the default is 1.
   4	Optional: Specifies how many successful pod completions are needed to mark a job completed. For non-parallel jobs, leave unset. When unset, the default is 1. For parallel jobs with a fixed completion count, specify the number of completions. For parallel jobs with a work queue, leave unset. When unset the default is the value of the parallelism parameter.
   5	Specifies the template for the pod the controller creates.
   6	Optional: Specifies the maximum number of replicas when scaling up. The default is 100.
   7	Optional: Specifies the interval in seconds to check each trigger on. The default is 30.
   8	Optional: Specifies the number of successful finished jobs should be kept. The default is 100.
   9	Optional: Specifies how many failed jobs should be kept. The default is 100.
   10	Optional: Specifies the name of the container in the target resource, from which the custom autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].
   11	Optional: Specifies whether existing jobs are terminated whenever a scaled job is being updated: default: The autoscaler terminates an existing job if its associated scaled job is updated. The autoscaler recreates the job with the latest specs. gradual: The autoscaler does not terminate an existing job if its associated scaled job is updated. The autoscaler creates new jobs with the latest specs.
   12	Optional: Specifies a scaling strategy: default, custom, or accurate. The default is default. For more information, see the link in the “Additional resources” section that follows.
   13	Specifies the trigger to use as the basis for scaling, as described in the “Understanding the custom metrics autoscaler triggers” section.
   14	Optional: Specifies a trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.
   15	Optional: Specifies a cluster trigger authentication, as described in the “Creating a custom metrics autoscaler trigger authentication” section.

2. Create the custom metrics autoscaler:

   $ oc create -f <file-name>.yaml

Verification

• View the command output to verify that the custom metrics autoscaler was created:

  $ oc get scaledjob <scaled_job_name>

  Example output

  NAME        MAX   TRIGGERS     AUTHENTICATION              READY   ACTIVE    AGE
  scaledjob   100   prometheus   prom-triggerauthentication  True    True      8s

  Note the following fields in the output:

• TRIGGERS: Indicates the trigger, or scaler, that is being used.

• AUTHENTICATION: Indicates the name of any trigger authentication being used.

• READY: Indicates whether the scaled object is ready to start scaling:

  • If True, the scaled object is ready.

  • If False, the scaled object is not ready because of a problem in one or more of the objects you created.

• ACTIVE: Indicates whether scaling is taking place:

  • If True, scaling is taking place.

  • If False, scaling is not taking place because there are no metrics or there is a problem in one or more of the objects you created.

Additional resources

• Understanding custom metrics autoscaler trigger authentications

You can remove the custom metrics autoscaler from your OKD cluster. After removing the Custom Metrics Autoscaler Operator, remove other components associated with the Operator to avoid potential issues.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. In the OKD web console, click Operators → Installed Operators.

2. Switch to the openshift-keda project.

3. Remove the KedaController custom resource.

   1. Find the CustomMetricsAutoscaler Operator and click the KedaController tab.

   2. Find the custom resource, and then click Delete KedaController.

   3. Click Uninstall.

4. Remove the Custom Metrics Autoscaler Operator:

   1. Click Operators → Installed Operators.

   2. Find the CustomMetricsAutoscaler Operator and click the Options menu and select Uninstall Operator.

   3. Click Uninstall.

5. Optional: Use the OpenShift CLI to remove the custom metrics autoscaler components:

   1. Delete the custom metrics autoscaler CRDs:

      • clustertriggerauthentications.keda.sh

      • kedacontrollers.keda.sh

      • scaledjobs.keda.sh

      • scaledobjects.keda.sh

      • triggerauthentications.keda.sh

      $ oc delete crd clustertriggerauthentications.keda.sh kedacontrollers.keda.sh scaledjobs.keda.sh scaledobjects.keda.sh triggerauthentications.keda.sh

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings. However, there might be a few cluster roles that must be manually deleted.

   2. List any custom metrics autoscaler cluster roles:

      $ oc get clusterrole | grep keda.sh

   3. Delete the listed custom metrics autoscaler cluster roles. For example:

      $ oc delete clusterrole.keda.sh-v1alpha1-admin

   4. List any custom metrics autoscaler cluster role bindings:

      $ oc get clusterrolebinding | grep keda.sh

   5. Delete the listed custom metrics autoscaler cluster role bindings. For example:

      $ oc delete clusterrolebinding.keda.sh-v1alpha1-admin

6. Delete the custom metrics autoscaler project:

   $ oc delete project openshift-keda