SidecarSet

    Besides injection during Pod creation, SidecarSet controller also provides additional capabilities such as in-place Sidecar container image upgrade, mounting Sidecar volumes, etc. Basically, SidecarSet decouples the Sidecar container lifecycle management from the main container lifecycle management.

    The SidecarSet is preferable for managing stateless sidecar containers such as monitoring tools or operation agents.

    The file below describes a SidecarSet that contains a sidecar container named sidecar1:

    Create a SidecarSet based on the YAML file:

    1. kubectl apply -f sidecarset.yaml

    Create a Pod

    Create a pod that matches the sidecarset’s selector:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   labels:
    5.     app: nginx # matches the SidecarSet's selector
    6.   name: test-pod
    7. spec:
    8.   containers:
    9.   - name: app
    10.     image: nginx:1.15.1

    Create this pod and now you will find it’s injected with sidecar1:

    1. $ kubectl get pod
    2. NAME       READY   STATUS    RESTARTS   AGE
    3. test-pod   2/2     Running   0          118s

    In the meantime, the SidecarSet status updated:

    1. $ kubectl get sidecarset test-sidecarset -o yaml | grep -A4 status
    2. status:
    3.   matchedPods: 1
    4.   observedGeneration: 1
    5.   readyPods: 1
    6.   updatedPods: 1

    update sidecarSet’s sidecar container image=centos:7

    1. $ kubectl edit sidecarsets test-sidecarset
    3. # sidecarset.yaml
    4. apiVersion: apps.kruise.io/v1alpha1
    5. kind: SidecarSet
    6. metadata:
    7.   name: test-sidecarset
    8. spec:
    9.   containers:
    10.     - name: sidecar1
    11.       image: centos:7

    The Sidecar container in the pod has been updated to centos:7, and the pod and other containers have not been restarted.

    SidecarSet features

    A sample SidecarSet yaml looks like following:

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   selector:
    7.     matchLabels:
    8.       app: sample
    9.   containers:
    10.   - name: nginx
    11.     image: nginx:alpine
    12.   initContainers:
    13.   - name: init-container
    14.     image: busybox:latest
    15.     command: [ "/bin/sh", "-c", "sleep 5 && echo 'init container success'" ]
    16.   updateStrategy:
    17.     type: RollingUpdate
    18.   namespace: ns-1
    • spec.selector Select the POD that needs to be injected and updated by Label. MatchLabels and MatchExpressions are supported. Please refer to the details: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
    • spec.initContainers Define the pod.spec.initContainers[x] you need to inject, supporting the full K8S InitContainer field. Please refer to the details：
      • We will inject those containers by their name in ascending order
      • InitContainers only support injection and do not support POD in-place update
    • spec.updateStrategy sidecarSet update strategy, type indicates the upgrade method:
      • NotUpdate No updates, in this type only inject sidecar containers in pod
      • RollingUpdate Injection and rolling update, which contains a rich update strategy, will be described in more detail later
    • spec.namespace By default, sidecarset is cluster scope in k8s, that is, for all namespaces (except kube-system, kube-public). When spec.namespace field set, it only applies to pods of that namespace

    sidecar container injection

    The injection of sidecar containers happens at Pod creation time and only Pod spec is updated. The workload template spec will not be updated. In addition to the default K8s Container field, the following fields have been extended to injection:

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    5. spec:
    6.   selector:
    7.     matchLabels:
    8.       app: sample
    9.   containers:
    10.     # default K8s Container fields
    11.   - name: nginx
    12.     image: nginx:alpine
    13.     volumeMounts:
    14.     - mountPath: /nginx/conf
    15.       name: nginx.conf
    16.     # extended sidecar container fields
    17.     podInjectPolicy: BeforeAppContainer
    18.     shareVolumePolicy:
    19.       type: disabled | enabled
    20.     transferEnv:
    21.     - sourceContainerName: main
    22.       envName: PROXY_IP    
    23.   volumes:
    24.   - Name: nginx.conf
    25.     hostPath: /data/nginx/conf
    • podInjectPolicy Define where Containers are injected into pod.spec.containers
      • BeforeAppContainer(default) Inject into the front of the original pod containers
      • AfterAppContainer Inject into the backend of the original pod containers
    • data volume sharing
      • Share specific volumes: Use spec.volumes to define the volumes needed by Sidecar itself. See details：
      • Share pod containers volumes: If ShareVolumePolicy.type is enabled, the sidecar container will share the other container’s VolumeMounts in the pod(don’t contains the injected sidecar container)
    • Environment variable sharing
      • Environment variables can be fetched from another container through spec.containers[x].transferenv, and the environment variable named envName from the container named sourceContainerName is copied to this container

    injection pause

    FEATURE STATE: Kruise v0.10.0

    For existing SidecarSets，users can pause sidecar injection by setting spec.injectionStrategy.paused=true：

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   ... ...
    7.   injectionStrategy:
    8.     paused: true

    imagePullSecrets

    FEATURE STATE: Kruise v0.10.0

    Users can use private images in SidecarSet by configuring spec.imagePullSecrets. SidecarSet will inject it in to Pods at injection stage.

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   ... ....
    7.   imagePullSecrets:
    8.   - name: my-secret

    Note: Users must ensure that the corresponding have already existed in the namespaces where Pods need to pull the private images. Otherwise, pulling private images will not succeed.

    Sidecarset not only supports the in-place update of Sidecar container, but also provides a very rich upgrade strategy.

    partition

    Partition is the desired number or percent of Pods in old revisions, defaults to 0. This field does NOT imply any update order.

    When partition is set during update:

    • If it is a number: (replicas - partition) number of pods will be updated with the new version.
    • If it is a percent: (replicas * (100% - partition)) number of pods will be updated with the new version.
    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   # ...
    7.   updateStrategy:
    8.     type: RollingUpdate
    9.     partition: 90

    Assuming that the number of PODs associated with this Sidecarset is 100, this upgrade will only upgrade 10 pods to latest and keep 90 pods old versions.

    MaxUnavailable

    MaxUnavailable is the maximum number of PODs that are unavailable at the same time that is guaranteed during the Posting process. The default value is 1.

    The user can set it to either an absolute value or a percentage (the percentage is calculated by the controller as the cardinality of the selected pod to calculate the absolute value behind one).

    Note that maxUnavailable and partition are not necessarily related. For example:

    • When {matched pod}=100,partition=50,maxUnavailable=10, the controller will update 50 PODS to the new version, and only 10 PODS will be updated at the same time, until the 50 updated is completed.
    • When {matched pod}=100,partition=80,maxUnavailable=30, the controller will update 20 PODS to the new version, because the maxUnavailable number is 30, so the 20 PODS will be updated simultaneously.

    Pause

    A user can pause the release by setting pause to true, and the injection capability will remain for newly created, expanded PODS, while updated PODS will remain the updated version, and those that have not been updated will be paused.

    1. apiVersion: apps.kruise.io/v1alpha1
    2. metadata:
    4. spec:
    5.   # ...
    6.   updateStrategy:
    7.     type: RollingUpdate
    8.     paused: true

    Selector

    For businesses that have Canary update requirements, this can be done through Strategy.selector filed. First: take the canary updated pods on fixed labels [canary. Release] = true, second fix the strategy.selector.MatchLabels to select the pod

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   # ...
    7.   updateStrategy:
    8.     type: RollingUpdate
    9.     selector:
    10.       matchLabels:
    11.       - canary.release: true

    sidecarset update order

    • The PODs of upgrade is sorted by default to ensure the same order of multiple upgrades
    • The default priority is (the smaller the higher the priority)： unscheduled < scheduled, pending < unknown < running, not-ready < ready, newer pods < older pods
    • scatter order

    scatter

    The scatter policy allows users to define the scatters of PODs that conform to certain tags throughout the publishing process. For example, if a SidecarSet manages 10 PODS, if there are 3 PODS below with the tag foo=bar, and the user sets this tag in the shatter policy, then these 3 PODS will be published in the 1st, 6th, and 10th positions.

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: sidecarset
    5. spec:
    6.   # ...
    7.   updateStrategy:
    8.     type: RollingUpdate
    9.     scatterStrategy:
    10.     - key: foo
    11.       value: bar

    FEATURE STATE: Kruise v0.9.0

    SidecarSet’s in-place upgrade will stop the container of old version first and then create the container of new version. Such method is more suitable for sidecar containers that cannot affects service availability, e.g. logging collector.

    But for many proxy or runtime sidecar containers, e.g. Istio Envoy, this upgrade method is problematic. Envoy, as a proxy container in the Pod, proxies all the traffic, and if restarted directly, the availability of service is affected. Complex grace termination and coordination is required if one need to upgrade envoy sidecar independently of the application container. So we provide a new solution for such sidecar container upgrade.

    1. apiVersion: apps.kruise.io/v1alpha1
    2. kind: SidecarSet
    3. metadata:
    4.   name: hotupgrade-sidecarset
    5. spec:
    6.   selector:
    7.     matchLabels:
    8.       app: hotupgrade
    9.   containers:
    10.   - name: sidecar
    11.     image: openkruise/hotupgrade-sample:sidecarv1
    12.     imagePullPolicy: Always
    13.     lifecycle:
    14.       postStart:
    15.         exec:
    16.           command:
    17.           - /bin/sh
    18.           - /migrate.sh
    19.     upgradeStrategy:
    20.       upgradeType: HotUpgrade
    21.       hotUpgradeEmptyImage: openkruise/hotupgrade-sample:empty
    • upgradeType: HotUpgrade indicates hot upgrade for stateful sidecar container.
    • hotUpgradeEmptyImage: when upgradeType=HotUpgrade, user needs to provide an empty container for hot upgrades. hotUpgradeEmptyImage has the same configuration as the sidecar container, for example: command, lifecycle, probe, etc, but it doesn’t do anything.
    • lifecycle.postStart: State Migration, the process completes the state migration of stateful container, which needs to be provided by the sidecar image developer.

    Hot upgrade consists of the following two processes:

    • inject hot upgrade sidecar containers
    • in-place hot upgrade sidecar container

    Inject Containers

    When the sidecar container upgradeStrategy=HotUpgrade, the SidecarSet Webhook will inject two containers into the Pod:

    1. {}-1: as shown in the figure below: envoy-1, the container run the actual working sidecar container, such as envoy:1.16.0
    2. {sidecarContainer.name}-2: as shown in the figure below: envoy-2, the container run the hot upgrade empty container, and it doesn’t have to deal with any real logic, as long as it stays in place, such as empty:1.0

    Hot Upgrade

    The SidecarSet Controller breaks down the hot upgrade pgrocess of the sidecar container into three steps:

    1. Upgrade: upgrade the empty container to the new version of the sidecar container, such as envoy-2.Image = envoy:1.17.0
    2. Migration: the process completes the state migration of stateful container, which needs to be provided by the sidecar image developer. PostStartHook completes the migration of the above process. (Note: PostStartHook must block during the migration, and exit when migration complete.)
    3. Reset: the step resets the old version sidecar container into empty container, such as envoy-1.Image = empty:1.0

    The above is the complete hot upgrade process. If a Pod needs to be hot upgraded several times, the above three steps can be repeated.

    sidecarset hotupgrade

    Migration Demo

    The SidecarSet hot upgrade mechanism not only completes the switching between mesh containers，but also provides a coordination mechanism for old and new versions. Yet this is only the first step of a long journey. The mesh container also needs to provide a PostStartHook script to complete the hot migration of the mesh service itself (the above Migration process), such as Envoy hot restart and Mosn lossless restart. To facilitate a better understanding of the Migration process, a migration demo is provided below the kruise repository:

    For design documentation, please refer to: proposals sidecarset hot upgrade

    Currently known cases that utilize the SidecarSet hot upgrade mechanism:

    • implements lossless upgrade of Data Plane in Service Mesh.

    SidecarSet Status

    When upgrading sidecar containers with a SidecarSet, you can observe the process of upgrading through SidecarSet.Status

    1. # kubectl describe sidecarsets sidecarset-example
    2. Name:         sidecarset-example
    3. Kind:         SidecarSet
    4. Status:
    5.   Matched Pods:         10  # The number of PODs injected and managed by the Sidecarset
    6.   Updated Pods:         5   # 5 PODs have been updated to the container version in the latest SidecarSet
    7.   Updated Ready Pods:   3   # Updated Pods && Ready Pods number