Locality failover

Before proceeding, be sure to complete the steps under before you begin.

In this task, you will use the Sleep pod in region1.zone1 as the source of requests to the HelloWorld service. You will then trigger failures that will cause failover between localities in the following sequence:

Locality failover sequence

Internally, Envoy priorities are used to control failover. These priorities will be assigned as follows for traffic originating from the Sleep pod (in region1 zone1):

Apply a DestinationRule that configures the following:

for the HelloWorld service. This is required in order for failover to function properly. In particular, it configures the sidecar proxies to know when endpoints for a service are unhealthy, eventually triggering a failover to the next locality.
Failover policy between regions. This ensures that failover beyond a region boundary will behave predictably.

Verify traffic stays in `region1.zone1`

Call the HelloWorld service from the Sleep pod:


  "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
  app=sleep -o jsonpath='{.items[0].metadata.name}')" \
  -- curl -sSL helloworld.sample:5000/hello
Hello version: region1.zone1, instance: helloworld-region1.zone1-86f77cd7b-cpxhv

Verify that the version in the response is region1.zone.

Next, trigger a failover to region1.zone2. To do this, you drain the Envoy sidecar proxy for HelloWorld in region1.zone1:

Call the HelloWorld service from the Sleep pod:

$ kubectl exec --context="${CTX_R1_Z1}" -n sample -c sleep \
  "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
  app=sleep -o jsonpath='{.items[0].metadata.name}')" \
  -- curl -sSL helloworld.sample:5000/hello

The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region1.zone2.

Failover to `region2.zone3`

Now trigger a failover to region2.zone3. As you did previously, configure the HelloWorld in to fail when called:

Call the HelloWorld service from the Sleep pod:

$ kubectl exec --context="${CTX_R1_Z1}" -n sample -c sleep \
  "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
  app=sleep -o jsonpath='{.items[0].metadata.name}')" \
  -- curl -sSL helloworld.sample:5000/hello
Hello version: region2.zone3, instance: helloworld-region2.zone3-86f77cd7b-cpxhv

The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region2.zone3.

Now trigger a failover to region3.zone4. As you did previously, configure the HelloWorld in region2.zone3 to fail when called:

Call the HelloWorld service from the Sleep pod:

$ kubectl exec --context="${CTX_R1_Z1}" -n sample -c sleep \
  "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
  app=sleep -o jsonpath='{.items[0].metadata.name}')" \
  -- curl -sSL helloworld.sample:5000/hello

The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region3.zone4.

Congratulations! You successfully configured locality failover!