Using Ingress-Nginx for Grayscale Release

    In Ingress-Nginx (v0.21.0), a new Canary feature has been introduced that can be used to configure multiple backend services for gateway portals, as well as to control traffic distribution between multiple backend services using specified annotations. . In , KubeSphere upgraded the Ingress Controller version to 0.24.1 to support Ingress-Nginx-based grayscale publishing.

    In the previous article, several application scenarios of grayscale publishing have been introduced in detail. This article will directly introduce and demonstrate the implementation of grayscale publishing based on KubeSphere using Ingress and Ingress Controller.

    Based on the , KubeSphere implements the project’s gateway as a proxy for external traffic and a reverse proxy for each service in the project. Ingress-Nginx supports the configuration of Ingress Annotations to implement grayscale publishing and testing in different scenarios. It can meet the business scenarios of Canary Publishing, Blue-Green Deployment, and A/B Testing.

    Nginx Annotations supports the following four Canary rules:

    • ：Flow segmentation based on Request Header for grayscale publishing and A/B testing. When the Request Header is set to always , the request will be sent to the Canary version all the time. When set to never, the request will not be sent to the Canary entry. For any other Header value, the Header will be ignored and the request is prioritized by comparison with other Canary rules.
    • nginx.ingress.kubernetes.io/canary-by-header-value：The value of the Request Header to match to tell Ingress to route the request to the service specified in Canary Ingress. When the Request Header is set to this value, it will be routed to the Canary entry. This rule allows the user to customize the value of the Request Header and must be used with the previous annotation (ie: canary-by-header).
    • nginx.ingress.kubernetes.io/canary-weight：Service-weighted traffic segmentation for blue-green deployments, weight range 0 - 100 Route requests to services specified in Canary Ingress. A weight of 0 means that the Canary Rules will not send any requests to the services of the Canary portal. A weight of 100 means that all requests will be sent to the Canary portal.
    • nginx.ingress.kubernetes.io/canary-by-cookie：Cookie-based traffic segmentation for grayscale publishing and A/B testing. A cookie that tells Ingress to route requests to the service specified in Canary Ingress. When the value of cookie is set to always , it will be routed to the Canary portal; when set to never , the request will not be sent to the Canary entry; for any other value, the cookie is ignored and the request is prioritized against other Canary rules.

    Note: The Canary Rules are sorted in order of priority: canary-by-header - > canary-by-cookie - > canary-weight

    The above four annotation rules can be divided into the following two categories:

    • Weight-based Canary rule

    • Canary rules based on user requests

    Ingress-Nginx for Grayscale Release - 图2

    Step 1: Create Project and the application of Production version

    1.1. Create a business space (spacespace) and a project (namespace) in KubeSphere. You can refer to Getting Started with Multi-tenant Management. A sample project has been created as follows.

    1.2. To create an application easily, create a workload and service in your project by edit yaml , or use the toolbox in the bottom right corner of KubeSphere to open web kubectl and create a Production version of the application with the following command and yaml file and expose it to the cluster access. Create a deployment and service for the Production version as follows.

    Ingress-Nginx for Grayscale Release - 图4

    The yaml file used is as follows:

    production. yaml

    1. apiVersion: extensions/v1beta1
    2. kind: Deployment
    3. metadata:
    4.   name: production
    5. spec:
    6.   replicas: 1
    7.   selector:
    8.     matchLabels:
    9.       app: production
    10.   template:
    11.     metadata:
    12.       labels:
    13.         app: production
    14.     spec:
    15.       containers:
    16.       - name: production
    17.         image: mirrorgooglecontainers/echoserver:1.10
    18.         - containerPort: 8080
    19.         env:
    20.           - name: NODE_NAME
    21.             valueFrom:
    22.               fieldRef:
    23.                 fieldPath: spec.nodeName
    24.           - name: POD_NAME
    25.             valueFrom:
    27.                 fieldPath: metadata.name
    28.           - name: POD_NAMESPACE
    29.             valueFrom:
    30.               fieldRef:
    31.                 fieldPath: metadata.namespace
    32.           - name: POD_IP
    33.             valueFrom:
    34.               fieldRef:
    35.                 fieldPath: status.podIP
    37. ---
    39. apiVersion: v1
    40. kind: Service
    41. metadata:
    42.   name: production
    43.   labels:
    44.     app: production
    45. spec:
    46.   ports:
    47.   - port: 80
    48.     targetPort: 8080
    49.     protocol: TCP
    50.     name: http
    51.     app: production

    The yaml file used is as follows:

    production.ingress

    1. apiVersion: extensions/v1beta1
    2. kind: Ingress
    3. metadata:
    4.   name: production
    5.   annotations:
    6.     kubernetes.io/ingress.class: nginx
    7. spec:
    8.   rules:
    9.   - host: kubesphere.io
    10.     http:
    11.       paths:
    12.       - backend:
    14.           servicePort: 80

    2.1. At this point, you can see all the resources under the ingress-demo project under the enterprise space demo-workspace of the KubeSphere UI.

    Deployment

    Service

    Ingress-Nginx for Grayscale Release - 图6

    Ingress

    2.2. Access the Production version of the application to ensure that the current project has opened the gateway, open the gateway under the external network access, the type is NodePort.

    Ingress-Nginx for Grayscale Release - 图8

    2.3. Access the Production version of the app below.

    Step 3: Create a Canary version

    Refer to the production.yaml file of the above Production version, and then create a Canary version of the application, including a Canary version of deployment and service (for demonstration, just replace the keyword production in the production.yaml deployment and service production to canary. The actual scenario may involve business code changes).

    A typical application scenario for weight-based traffic segmentation is a Blue-green deployment, which can be achieved by setting the weight to 0 or 100. For example, you can set the Green version to the main part and the Blue version’s entry to Canary. Initially, the weight is set to 0, so traffic is not proxied to the Blue version. Once the new version has been tested and verified successfully, you can set the weight of the Blue version to 100, which means that all traffic goes from Green to Blue.

    4.1. Use the following canary.ingress yaml file to create an application routing (Ingress) based on the weighted Canary version.

    1. apiVersion: extensions/v1beta1
    2. kind: Ingress
    3. metadata:
    4.   name: canary
    5.   annotations:
    6.     kubernetes.io/ingress.class: nginx
    7.     nginx.ingress.kubernetes.io/canary: "true"
    8.     nginx.ingress.kubernetes.io/canary-weight: "30"
    9. spec:
    10.   rules:
    11.   - host: kubesphere.io
    12.     http:
    13.       paths:
    14.       - backend:
    15.           serviceName: canary
    16.           servicePort: 80

    4.2. Access the domain name of the app.

    Note: After the application’s Canary version is based on the weight (30%) for traffic segmentation, the probability of accessing the Canary version is close to 30%, and the traffic ratio may have a small range of fluctuations.

    Ingress-Nginx for Grayscale Release - 图10

    4.3. The classic application scenes based on Request Header’s traffic segmentation are greyscale release or A/B test scenes. Refer to the screenshot below. Add an annotation of nginx.ingress.kubernetes.io/canary-by-header: canary for Ingress in KubeSphere’s Canary edition. (The annotation value here can be random.) Realize the current Ingress and segregate the traffic based on the Ingress.

    4.4. Add a different Header value to the request and access the domain name of the app again.

    Note:

    For examples, as mentioned in the opening paragraph, when the Request Header is set to never or always, the request will not or always be sent or always sent to the Canary version; For any other Header value, the Header is ignored and the request is prioritized by comparison with other Canary rules (the second request has a 30% weight as the first priority).

    Ingress-Nginx for Grayscale Release - 图12

    4.5. You can add an nginx.ingress.kubernetes.io/canary-by-header-value: user-value to the previous annotation (that is, canary-by-header). Used to notify Ingress to route requests to the service specified in Canary Ingress.

    4.6. The application’s domain name is accessed as follows. When the Request Header satisfies this value, all requests are routed to the Canary version (this rule allows the user to customize the value of the Request Header).

    Ingress-Nginx for Grayscale Release - 图14

    Summary

    The grayscale release can ensure the stability of the overall system. When the initial grayscale is used, the new version can be tested, found and adjusted to ensure its influence. This article demonstrates and demonstrates the use of Ingress and Ingress Controller for grayscale publishing based on Kubeel, and details the four Annotations of Ingress-Nginx. Users who have not used Istio can also easily implement grayscale publishing with Ingress-Nginx. Released with the canary.