TCP Proxying and Protocol Detection

    In most cases, Linkerd can do this without configuration. To accomplish this, Linkerd performs protocol detection to determine whether traffic is HTTP or HTTP/2 (including gRPC). If Linkerd detects that a connection is HTTP or HTTP/2, Linkerd automatically provides HTTP-level metrics and routing.

    If Linkerd cannot determine that a connection is using HTTP or HTTP/2, Linkerd will proxy the connection as a plain TCP connection, applying mTLS and providing byte-level metrics as usual.

    (Note that HTTPS calls to or from meshed pods are treated as TCP, not as HTTP. Because the client initiates the TLS connection, Linkerd is not be able to decrypt the connection to observe the HTTP transactions.)

    Note

    If you are experiencing 10-second delays when establishing connections, you are likely running into a protocol detection timeout. This section will help you understand how to fix this.

    In some cases, Linkerd’s protocol detection will time out because it doesn’t see any bytes from the client. This situation is commonly encountered when using “server-speaks-first” protocols where the server sends data before the client does, such as SMTP, or protocols that proactively establish connections without sending data, such as Memcache. In this case, the connection will proceed as a TCP connection after a 10-second protocol detection delay.

    By default, Linkerd automatically marks the ports for some server-speaks-first protocol as opaque. Services that speak those protocols over the default ports to destinations inside the cluster do not need further configuration. Linkerd’s default list of opaque ports in the 2.11 release is 25 (SMTP), 587 (SMTP), 3306 (MySQL), 4444 (Galera), 5432 (Postgres), 6379 (Redis), 9300 (ElasticSearch), and 11211 (Memcache). Note that this may change in future releases.

    The following table contains common protocols that may require configuration.

    If you are using one of those protocols, follow this decision tree to determine which configuration you need to apply.

    • Is the protocol wrapped in TLS?
      • Yes: no configuration required.
      • Yes: is the port in Linkerd’s default list of opaque ports?
        • Yes: no configuration required.
      • No: mark port(s) as skip.

    You can use the annotation to mark a port as opaque. This instructions Linkerd to skip protocol detection for that port.

    This annotation can be set on a workload, service, or namespace. Setting it on a workload tells meshed clients of that workload to skip protocol detection for connections established to the workload, and tells Linkerd to skip protocol detection when reverse-proxying incoming connections. Setting it on a service tells meshed clients to skip protocol detection when proxying connections to the service. Set it on a namespace applies this behavior to all services and workloads in that namespace.

    Note

    Setting the opaque-ports annotation can be done by using the --opaque-ports flag when running . For example, for a MySQL database running on the cluster using a non-standard port 4406, you can use the commands:

    Note

    Multiple ports can be provided as a comma-delimited string. The values you provide will replace, not augment, the default list of opaque ports.

    Sometimes it is necessary to bypass the proxy altogether. For example, when connecting to a server-speaks-first destination that is outside of the cluster, there is no Service resource on which to set the config.linkerd.io/opaque-ports annotation.

    In this case, you can use the flag when running linkerd inject to configure resources to bypass the proxy entirely when sending to those ports. (Similarly, the flag will configure the resource to bypass the proxy for incoming connections to those ports.)

    Skipping the proxy can be useful for these situations, as well as for diagnosing issues, but otherwise should rarely be necessary.