Securing Your Application with mTLS

    Linkerd’s automatic mTLS is done in a way that’s completely transparent to the application. Of course, sometimes it’s helpful to be able to validate whether mTLS is in effect!

    Note

    Linkerd uses Kubernetes ServiceAccounts to define service identity. This requires that the automountServiceAccountToken feature (on by default) has not been disabled on the pods. See the Kubernetes service account documentation for more.

    To validate that mTLS is working, you can view a summary of the TCP connections between services that are managed by Linkerd using the command. For example:

    The output will look like:

    1. SRC          DST                      SRC_NS        DST_NS    SECURED
    2. prometheus   linkerd-controller       linkerd-viz   linkerd   √
    3. prometheus   linkerd-destination      linkerd-viz   linkerd   √
    4. prometheus   linkerd-proxy-injector   linkerd-viz   linkerd   √
    5. prometheus   linkerd-sp-validator     linkerd-viz   linkerd   √

    Instead of relying on an aggregate, it is also possible to watch the requests and responses in real time to understand what is getting mTLS’d. We can use the linkerd viz tap command to sample real time request data.

    1. linkerd viz -n linkerd tap deploy

    Note

    By default, the control plane resources are not tappable. After having installed the Viz extension (through linkerd viz install), you can enable tap on the control plane components simply by restarting them, which can be done with no downtime with kubectl -n linkerd rollout restart deploy. To enable tap on the Viz extension itself, issue kubectl -n linkerd-viz rollout restart deploy.

    Looking at the control plane specifically, there will be two main types of output.

    These are calls by the . As probes are initiated from the kubelet, which is not in the mesh, there is no identity and these requests are not mTLS’d, as denoted by the tls=no_tls_from_remote message.

    1. ireq id=2:1 proxy=in  src=10.42.0.31:55428 dst=10.42.0.22:9995 tls=true :method=GET :authority=10.42.0.22:9995 :path=/metrics
    2. rsp id=2:1 proxy=in  src=10.42.0.31:55428 dst=10.42.0.22:9995 tls=true :status=200 latency=1597µs
    3. end id=2:1 proxy=in  src=10.42.0.31:55428 dst=10.42.0.22:9995 tls=true duration=228µs response-length=2272B

    This connection comes from Prometheus, which in the mesh, so the request is automatically mTLS’d, as denoted by the tls=true output.

    The final way to validate mTLS is to look at raw network traffic within the cluster.

    Linkerd includes a debug sidecar that comes with a selection of commands that make it easier to verify and debug the service mesh itself. For example, with our , we can add the debug sidecar by running:

    2.   | kubectl apply -f -

    We can then establish a remote shell directly in the debug container of a pod in the voting service with:

    Once we’re inside the debug sidecar, the built-in tshark command can be used to inspect the raw packets on the network interface. For example:

    1. tshark -i any -d tcp.port==8080,ssl | grep -v 127.0.0.1
    1.   133 11.391540872    10.4.0.17 → 10.4.0.23    TCP 68 46766 → 4191 [ACK] Seq=557 Ack=3942 Win=1329 Len=0 TSval=3389590636 TSecr=1915605020
    2.   134 12.128190076    10.4.0.25 → 10.4.0.23    TLSv1.2 154 Application Data
    3.   140 12.129497053    10.4.0.23 → 10.4.0.25    TLSv1.2 149 Application Data
    4.   141 12.129534848    10.4.0.25 → 10.4.0.23    TCP 68 48138 → 8080 [ACK] Seq=1089 Ack=985 Win=236 Len=0 TSval=2234109459 TSecr=617799816
    5.   143 13.140288400    10.4.0.25 → 10.4.0.23    TLSv1.2 150 Application Data

    In this guide, we’ve provided several different ways to validate whether Linkerd has been able to automatically upgrade connections to mTLS. Note that there are several reasons why Linkerd may not be able to do this upgrade—see the “Caveats and future work” section of the Linkerd automatic mTLS documentation—so if you are relying on Linkerd for security purposes, this kind of validation can be instructive.