Using the Debug Sidecar

    In cases where you need network-level visibility into packets entering and leaving your application, Linkerd provides a debug sidecar with some helpful tooling. Similar to how proxy sidecar injection works, you add a debug sidecar to a pod by setting the annotation at pod creation time. For convenience, the linkerd inject command provides an --enable-debug-sidecar option that does this annotation for you.

    (Note that the set of containers in a Kubernetes pod is not mutable, so simply adding this annotation to a pre-existing pod will not work. It must be present at pod creation time.)

    The debug sidecar image contains , tcpdump, lsof, and iproute2. Once installed, it starts automatically logging all incoming and outgoing traffic with tshark, which can then be viewed with kubectl logs. Alternatively, you can use to access the container and run commands directly.

    to deploy the debug sidecar container to all pods in the voting service. (Note that there’s only one pod in this deployment, which will be recreated to do this–see the note about pod mutability above.)

    You can confirm that the debug container is running by listing all the containers in pods with the voting-svc label:

    1. kubectl get pods -n emojivoto -l app=voting-svc \
    2.   -o jsonpath='{.items[*].spec.containers[*].name}'

    Then, you can watch live tshark output from the logs by simply running:

    1. kubectl -n emojivoto exec -it \
    2.     -o jsonpath='{.items[0].metadata.name}') \
    3.   -c linkerd-debug -- tshark -i any -f "tcp" -V -Y "http.request"

    A real-world error message written by the proxy that the debug sidecar is effective in troubleshooting is a Connection Refused error like this one:

    In this case, the command can be modified to listen for traffic between the specific ports mentioned in the error, like this:

    1. kubectl -n emojivoto exec -it \
    2.   -o jsonpath='{.items[0].metadata.name}') \
    3.   -c linkerd-debug -- tshark -i any -f "tcp" -V \
    4.   -Y "(tcp.srcport == 4143 and tcp.dstport == 50416) or tcp.port == 8080"

    Be aware that there is a similar error with the message Connection reset by peer. This error is usually benign, if you do not see correlated errors or messages in your application log output. In this scenario, the debug container may not help to troubleshoot the error message.