Multi-cluster communication with StatefulSets

    This guide will walk you through installing and configuring Linkerd and the multi-cluster extension with support for headless services and will exemplify how a StatefulSet can be deployed in a target cluster. After deploying, we will also look at how to communicate with an arbitrary pod from the target cluster’s StatefulSet from a client in the source cluster. For a more detailed overview on how multi-cluster support for headless services work, check out multi-cluster communication.

    • Two Kubernetes clusters. They will be referred to as east and west with east being the “source” cluster and “west” the target cluster respectively. These can be in any cloud or local environment, this guide will make use of to configure two local clusters.
    • smallstep/CLI to generate certificates for Linkerd installation.
    • to install Linkerd.

    To help with cluster creation and installation, there is a demo repository available. Throughout the guide, we will be using the scripts from the repository, but you can follow along without cloning or using the scripts.

    To start our demo and see everything in practice, we will go through a multi-cluster scenario where a pod in an east cluster will try to communicate to an arbitrary pod from a west cluster.

    The first step is to clone the demo repository on your local machine.

    The second step consists of creating two k3d clusters named east and west, where the east cluster is the source and the west cluster is the target. When creating our clusters, we need a shared trust root. Luckily, the repository you have just cloned includes a handful of scripts that will greatly simplify everything.

    1. # create k3d clusters
    2. $ ./create.sh
    4. # list the clusters
    5. $ k3d cluster list
    6. NAME   SERVERS   AGENTS   LOADBALANCER
    7. east   1/1       0/0      true
    8. west   1/1       0/0      true

    Once our clusters are created, we will install Linkerd and the multi-cluster extension. Finally, once both are installed, we need to link the two clusters together so their services may be mirrored. To enable support for headless services, we will pass an additional --set "enableHeadlessServices=true flag to linkerd multicluster link. As before, these steps are automated through the provided scripts, but feel free to have a look!

    1. # Install Linkerd and multicluster, output to check should be a success
    2. $ ./install.sh
    4. # Next, link the two clusters together
    5. $ ./link.sh

    With our install steps out of the way, we can now focus on our pod-to-pod communication. First, we will deploy our pods and services:

    • We will mesh the default namespaces in east and west.
    • In west, we will deploy an nginx StatefulSet with its own headless service, nginx-svc.

    • In east, our script will deploy a curl pod that will then be used to curl the nginx service.

    Before we go further, let’s have a look at the endpoints object for the nginx-svc:

    1. $ kubectl --context=k3d-west get endpoints nginx-svc -o yaml
    3. subsets:
    4. - addresses:
    5.   - hostname: nginx-set-0
    6.     ip: 10.42.0.31
    7.     nodeName: k3d-west-server-0
    8.       kind: Pod
    9.       name: nginx-set-0
    10.       namespace: default
    11.       resourceVersion: "114743"
    12.       uid: 7049f1c1-55dc-4b7b-a598-27003409d274
    13.   - hostname: nginx-set-1
    14.     ip: 10.42.0.32
    15.     nodeName: k3d-west-server-0
    16.     targetRef:
    17.       kind: Pod
    18.       name: nginx-set-1
    19.       namespace: default
    20.       resourceVersion: "114775"
    21.       uid: 60df15fd-9db0-4830-9c8f-e682f3000800
    22.   - hostname: nginx-set-2
    23.     ip: 10.42.0.33
    24.     nodeName: k3d-west-server-0
    25.     targetRef:
    26.       kind: Pod
    27.       name: nginx-set-2
    28.       namespace: default
    29.       resourceVersion: "114808"
    30.       uid: 3873bc34-26c4-454d-bd3d-7c783de16304

    We can see, based on the endpoints object that the service has three endpoints, with each endpoint having an address (or IP) whose hostname corresponds to a StatefulSet pod. If we were to do a curl to any of these endpoints directly, we would get an answer back. We can test this out by applying the curl pod to the west cluster:

    1. $ kubectl --context=k3d-west apply -f east/curl.yml
    2. $ kubectl --context=k3d-west get pods
    3. NAME                    READY   STATUS            RESTARTS   AGE
    4. nginx-set-0             2/2     Running           0          5m8s
    5. nginx-set-1             2/2     Running           0          4m58s
    7. curl-56dc7d945d-s4n8j   0/2     PodInitializing   0          4s
    8. $ kubectl --context=k3d-west exec -it curl-56dc7d945d-s4n8j -c curl -- bin/sh
    9. /$ # prompt for curl pod

    If we now curl one of these instances, we will get back a response.

    Now, let’s do the same, but this time from the east cluster. We will first export the service.

    1. $ kubectl --context=k3d-west label service nginx-svc mirror.linkerd.io/exported="true"
    2. service/nginx-svc labeled
    4. $ kubectl --context=k3d-east get services
    5. NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    6. kubernetes         ClusterIP   10.43.0.1       <none>        443/TCP   20h
    7. nginx-svc-west     ClusterIP   None            <none>        80/TCP    29s
    8. nginx-set-0-west   ClusterIP   10.43.179.60    <none>        80/TCP    29s
    9. nginx-set-1-west   ClusterIP   10.43.218.18    <none>        80/TCP    29s
    10. nginx-set-2-west   ClusterIP   10.43.245.244   <none>        80/TCP    29s
    1. $ kubectl --context=k3d-east get endpoints nginx-svc-west -o yaml
    2. subsets:
    3. - addresses:
    4.   - hostname: nginx-set-0
    5.     ip: 10.43.179.60
    6.   - hostname: nginx-set-1
    7.     ip: 10.43.218.18
    8.   - hostname: nginx-set-2
    9.     ip: 10.43.245.244

    This is what we outlined at the start of the tutorial. Each pod from the target cluster (west), will be mirrored as a clusterIP service. We will see in a second why this matters.

    As you can see, we get the same response back! But, nginx is in a different cluster. So, what happened behind the scenes?

    1. When we mirrored the headless service, we created a clusterIP service for each pod. Since services create DNS records, naming each endpoint with the hostname from the target gave us these pod FQDNs (nginx-set-0.(...).cluster.local).
    2. Curl resolved the pod DNS name to an IP address. In our case, this IP would be 10.43.179.60.
    3. Once the request is in-flight, the linkerd2-proxy intercepts it. It looks at the IP address and associates it with our clusterIP service. The service itself points to the gateway, so the proxy forwards the request to the target cluster gateway. This is the usual multi-cluster scenario.
    4. The gateway in the target cluster looks at the request and looks-up the original destination address. In our case, since this is an “endpoint mirror”, it knows it has to go to nginx-set-0.nginx-svc in the same cluster.
    5. The request is again forwarded by the gateway to the pod, and the response comes back.

    And that’s it! You can now send requests to pods across clusters. Querying any of the 3 StatefulSet pods should have the same results.

    Note

    To mirror a headless service as headless, the service’s endpoints must also have at least one named address (e.g a hostname for an IP), otherwise, there will be no endpoints to mirror so the service will be mirrored as clusterIP. A headless service may under normal conditions also be created without exposing a port; the mulit-cluster service-mirror does not support this, however, since the lack of ports means we cannot create a service that passes Kubernetes validation.

    To clean-up, you can remove both clusters entirely using the k3d CLI:

    1. $ k3d cluster delete east
    2. cluster east deleted
    3. $ k3d cluster delete west
    4. cluster west deleted