Controlling pod placement on nodes using node affinity rules

    In OKD node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on the nodes and label selectors specified in pods.

    Node affinity allows a pod to specify an affinity towards a group of nodes it can be placed on. The node does not have control over the placement.

    For example, you could configure a pod to only run on a node with a specific CPU or in a specific availability zone.

    There are two types of node affinity rules: required and preferred.

    Required rules must be met before a pod can be scheduled on a node. Preferred rules specify that, if the rule is met, the scheduler tries to enforce the rules, but does not guarantee enforcement.

    You configure node affinity through the spec file. You can specify a required rule, a preferred rule, or both. If you specify both, the node must first meet the required rule, then attempts to meet the preferred rule.

    The following example is a Pod spec with a rule that requires the pod be placed on a node with a label whose key is e2e-az-NorthSouth and whose value is either e2e-az-North or e2e-az-South:

    Example pod configuration file with a node affinity required rule

    1The stanza to configure node affinity.
    2Defines a required rule.
    3The key/value pair (label) that must be matched to apply the rule.
    4The operator represents the relationship between the label on the node and the set of values in the matchExpression parameters in the Pod spec. This value can be In, NotIn, Exists, or DoesNotExist, Lt, or Gt.

    The following example is a node specification with a preferred rule that a node with a label whose key is e2e-az-EastWest and whose value is either e2e-az-East or e2e-az-West is preferred for the pod:

    Example pod configuration file with a node affinity preferred rule

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: with-node-affinity
    5. spec:
    6.   affinity:
    7.     nodeAffinity: (1)
    8.       preferredDuringSchedulingIgnoredDuringExecution: (2)
    9.       - weight: 1 (3)
    10.         preference:
    11.           matchExpressions:
    12.           - key: e2e-az-EastWest (4)
    13.             operator: In (5)
    14.             values:
    15.             - e2e-az-East (4)
    16.             - e2e-az-West (4)
    17.   containers:
    18.   - name: with-node-affinity
    19.     image: docker.io/ocpqe/hello-pod
    1The stanza to configure node affinity.
    2Defines a preferred rule.
    3Specifies a weight for a preferred rule. The node with highest weight is preferred.
    4The key/value pair (label) that must be matched to apply the rule.
    5The operator represents the relationship between the label on the node and the set of values in the matchExpression parameters in the Pod spec. This value can be In, NotIn, Exists, or DoesNotExist, Lt, or Gt.

    There is no explicit node anti-affinity concept, but using the NotIn or DoesNotExist operator replicates that behavior.

    Configuring a required node affinity rule

    Required rules must be met before a pod can be scheduled on a node.

    Procedure

    The following steps demonstrate a simple configuration that creates a node and a pod that the scheduler is required to place on the node.

    1. Add a label to a node using the oc label node command:

      1. $ oc label node node1 e2e-az-name=e2e-az1

      You can alternatively apply the following YAML to add the label:

      1. kind: Node
      2. apiVersion: v1
      3. metadata:
      4.   name: <node_name>
      5.   labels:
      6.     e2e-az-name: e2e-az1

    2. In the Pod spec, use the nodeAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter:

      2. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the operator In to require the label to be in the node:

        Example output

        1. spec:
        2.   affinity:
        3.     nodeAffinity:
        4.       requiredDuringSchedulingIgnoredDuringExecution:
        5.         nodeSelectorTerms:
        6.         - matchExpressions:
        7.           - key: e2e-az-name
        9.             values:
        10.             - e2e-az1
        11.             - e2e-az2

    3. Create the pod:

      1. $ oc create -f e2e-az2.yaml

    Preferred rules specify that, if the rule is met, the scheduler tries to enforce the rules, but does not guarantee enforcement.

    Procedure

    The following steps demonstrate a simple configuration that creates a node and a pod that the scheduler tries to place on the node.

    1. Add a label to a node using the oc label node command:

      1. $ oc label node node1 e2e-az-name=e2e-az3

    2. In the Pod spec, use the nodeAffinity stanza to configure the preferredDuringSchedulingIgnoredDuringExecution parameter:

      1. Specify a weight for the node, as a number 1-100. The node with highest weight is preferred.

      2. Specify the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and value parameters as the label in the node:

        1. spec:
        2.   affinity:
        3.     nodeAffinity:
        4.       preferredDuringSchedulingIgnoredDuringExecution:
        5.       - weight: 1
        6.         preference:
        7.           matchExpressions:
        8.           - key: e2e-az-name
        9.             operator: In
        10.             - e2e-az3

      3. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the Operator In to require the label to be in the node.

    3. Create the pod.

    Sample node affinity rules

    The following examples demonstrate node affinity.

    The following example demonstrates node affinity for a node and pod with matching labels:

    • The Node1 node has the label zone:us:

      1. $ oc label node node1 zone=us

      You can alternatively apply the following YAML to add the label:

      1. kind: Node
      2. apiVersion: v1
      3. metadata:
      4.   name: <node_name>
      5.   labels:
      6.     zone: us

    • The pod-s1 pod has the zone and us key/value pair under a required node affinity rule:

      1. $ cat pod-s1.yaml

      Example output

      1. apiVersion: v1
      2. kind: Pod
      3. metadata:
      4.   name: pod-s1
      5. spec:
      6.   containers:
      7.     - image: "docker.io/ocpqe/hello-pod"
      8.       name: hello-pod
      9.   affinity:
      10.     nodeAffinity:
      11.       requiredDuringSchedulingIgnoredDuringExecution:
      12.         nodeSelectorTerms:
      13.           - matchExpressions:
      14.             - key: "zone"
      15.               operator: In
      16.               values:
      17.               - us

    • The pod-s1 pod can be scheduled on Node1:

      1. $ oc get pod -o wide

      Example output

      1. NAME     READY     STATUS       RESTARTS   AGE      IP      NODE
      2. pod-s1   1/1       Running      0          4m       IP1     node1

    Node affinity with no matching labels

    The following example demonstrates node affinity for a node and pod without matching labels:

    • The Node1 node has the label zone:emea:

      1. $ oc label node node1 zone=emea

    • The pod-s1 pod has the zone and us key/value pair under a required node affinity rule:

      1. $ cat pod-s1.yaml

      Example output

      1. apiVersion: v1
      2. kind: Pod
      3. metadata:
      4.   name: pod-s1
      5. spec:
      6.   containers:
      7.     - image: "docker.io/ocpqe/hello-pod"
      8.       name: hello-pod
      9.   affinity:
      10.     nodeAffinity:
      11.       requiredDuringSchedulingIgnoredDuringExecution:
      12.         nodeSelectorTerms:
      13.           - matchExpressions:
      14.             - key: "zone"
      16.               values:
      17.               - us

    • The pod-s1 pod cannot be scheduled on Node1:

      1. $ oc describe pod pod-s1

      Example output

      1. ...
      3. Events:
      4.  FirstSeen LastSeen Count From              SubObjectPath  Type                Reason
      5.  --------- -------- ----- ----              -------------  --------            ------
      6.  1m        33s      8     default-scheduler Warning        FailedScheduling    No nodes are available that match all of the following predicates:: MatchNodeSelector (1).

    By default, when you install an Operator, OKD installs the Operator pod to one of your worker nodes randomly. However, there might be situations where you want that pod scheduled on a specific node or set of nodes.

    The following examples describe situations where you might want to schedule an Operator pod to a specific node or set of nodes:

    • If an Operator requires a particular platform, such as amd64 or arm64

    • If an Operator requires a particular operating system, such as Linux or Windows

    • If you want Operators that work together scheduled on the same host or on hosts located on the same rack

    • If you want Operators dispersed throughout the infrastructure to avoid downtime due to network or hardware issues

    You can control where an Operator pod is installed by adding a node affinity constraints to the Operator’s Subscription object.

    The following examples show how to use node affinity to install an instance of the Custom Metrics Autoscaler Operator to a specific node in the cluster:

    Node affinity example that places the Operator pod on a specific node

    1. apiVersion: operators.coreos.com/v1alpha1
    2. metadata:
    3.   name: openshift-custom-metrics-autoscaler-operator
    4.   namespace: openshift-keda
    5. spec:
    6.   name: my-package
    7.   source: my-operators
    8.   sourceNamespace: operator-registries
    9.   config:
    10.     affinity:
    11.       nodeAffinity: (1)
    12.         requiredDuringSchedulingIgnoredDuringExecution:
    13.           nodeSelectorTerms:
    14.           - matchExpressions:
    15.             - key: kubernetes.io/hostname
    16.               operator: In
    17.               values:
    18.               - ip-10-0-163-94.us-west-2.compute.internal
    19.  ...
    1A node affinity that requires the Operator’s pod to be scheduled on a node named ip-10-0-163-94.us-west-2.compute.internal.

    Node affinity example that places the Operator pod on a node with a specific platform

    1. apiVersion: operators.coreos.com/v1alpha1
    2. kind: Subscription
    3. metadata:
    4.   name: openshift-custom-metrics-autoscaler-operator
    5.   namespace: openshift-keda
    6. spec:
    7.   name: my-package
    8.   source: my-operators
    9.   sourceNamespace: operator-registries
    10.   config:
    11.     affinity:
    12.       nodeAffinity: (1)
    13.         requiredDuringSchedulingIgnoredDuringExecution:
    14.           nodeSelectorTerms:
    15.           - matchExpressions:
    16.             - key: kubernetes.io/arch
    17.               operator: In
    18.               values:
    19.               - arm64
    20.             - key: kubernetes.io/os
    21.               operator: In
    22.               values:
    23.               - linux
    1A node affinity that requires the Operator’s pod to be scheduled on a node with the kubernetes.io/arch=arm64 and kubernetes.io/os=linux labels.

    Procedure

    To control the placement of an Operator pod, complete the following steps:

    1. Install the Operator as usual.

    2. If needed, ensure that your nodes are labeled to properly respond to the affinity.

    3. Edit the Operator Subscription object to add an affinity:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: Subscription
      3. metadata:
      4.   name: openshift-custom-metrics-autoscaler-operator
      5.   namespace: openshift-keda
      6. spec:
      7.   name: my-package
      8.   source: my-operators
      9.   sourceNamespace: operator-registries
      10.   config:
      11.     affinity: (1)
      12.       nodeAffinity:
      13.         requiredDuringSchedulingIgnoredDuringExecution:
      14.           nodeSelectorTerms:
      15.           - matchExpressions:
      16.             - key: kubernetes.io/hostname
      17.               operator: In
      18.               values:
      19.               - ip-10-0-185-229.ec2.internal
      20.  ...

    Verification

    • To ensure that the pod is deployed on the specific node, run the following command:

      Example output

      2. custom-metrics-autoscaler-operator-5dcc45d656-bhshg   1/1     Running   0          50s   10.131.0.20   ip-10-0-185-229.ec2.internal   <none>           <none>

    Additional resources