Mirroring

Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

In this task, you will first force all traffic to of a test service. Then, you will apply a rule to mirror a portion of traffic to v2.

• Set up Istio by following the instructions in the Installation guide.

• Start by deploying two versions of the service that have access logging enabled:

  httpbin-v1:

  httpbin-v2:

  $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: httpbin-v2
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: httpbin
        version: v2
    template:
      metadata:
        labels:
          app: httpbin
          version: v2
      spec:
        containers:
        - image: docker.io/kennethreitz/httpbin
          imagePullPolicy: IfNotPresent
          name: httpbin
          command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
          ports:
          - containerPort: 80
  EOF

  httpbin Kubernetes service:

  $ kubectl create -f - <<EOF
  apiVersion: v1
  metadata:
    name: httpbin
    labels:
      app: httpbin
  spec:
    ports:
    - name: http
      targetPort: 80
    selector:
      app: httpbin
  EOF

• Start the sleep service so you can use curl to provide load:

  sleep service:

  $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: sleep
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: sleep
    template:
      metadata:
        labels:
          app: sleep
      spec:
        containers:
        - name: sleep
          image: tutum/curl
          command: ["/bin/sleep","infinity"]
          imagePullPolicy: IfNotPresent
  EOF

By default Kubernetes load balances across both versions of the httpbin service. In this step, you will change that behavior so that all traffic goes to v1.

1. Send some traffic to the service:

   $ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
   $ kubectl exec -it "$SLEEP_POD" -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool
   {
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin:8000",
       "User-Agent": "curl/7.35.0",
       "X-B3-Sampled": "1",
       "X-B3-Spanid": "eca3d7ed8f2e6a0a",
       "X-B3-Traceid": "eca3d7ed8f2e6a0a",
       "X-Ot-Span-Context": "eca3d7ed8f2e6a0a;eca3d7ed8f2e6a0a;0000000000000000"
     }
   }

2. Check the logs for v1 and v2 of the httpbin pods. You should see access log entries for v1 and none for v2:

   $ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
   $ kubectl logs -f "$V1_POD" -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"

   $ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
   $ kubectl logs -f "$V2_POD" -c httpbin
   <none>

1. Change the route rule to mirror traffic to v2:

   This route rule sends 100% of the traffic to v1. The last stanza specifies that you want to mirror to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

   Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.

   You can use the mirror_percent field to mirror a fraction of the traffic, instead of mirroring all requests. If this field is absent, for compatibility with older versions, all traffic will be mirrored.

2. Send in traffic:

   $ kubectl exec -it "$SLEEP_POD" -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool

   Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

   $ kubectl logs -f "$V1_POD" -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
   127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"

   $ kubectl logs -f "$V2_POD" -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

1. Remove the rules:

2. $ kubectl delete deploy httpbin-v1 httpbin-v2 sleep
   

Traffic Mirroring with Istio for Testing in Production

An introduction to safer, lower-risk deployments and release to production.

Configure the IBM Cloud Kubernetes Service Application Load Balancer to direct traffic to the Istio Ingress gateway with mutual TLS.

Multicluster Istio configuration and service discovery using Admiral

Automating Istio configuration for Istio deployments (clusters) that work as a single mesh.

Configure Istio ingress gateway to act as a proxy for external services.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

Comparison of alternative solutions to control egress traffic including performance considerations.