Overview

    • Streaming delivery of xDS API updates via gRPC. This reduces resource requirements and can lower the update latency.
    • A new REST-JSON API in which the JSON/YAML formats are derived mechanically via the .
    • Delivery of updates via the filesystem, REST-JSON or gRPC endpoints.
    • Advanced load balancing through an extended endpoint assignment API and load and resource utilization reporting to management servers.
    • Stronger consistency and ordering properties when needed. The v2 APIs still maintain a baseline eventual consistency model.

    See the for further details on aspects of v2 message exchange between Envoy and the management server.

    To use the v2 API, it’s necessary to supply a bootstrap configuration file. This provides static server configuration and configures Envoy to access dynamic configuration if needed. This is supplied on the command-line via the flag, i.e.:

    where the extension reflects the underlying v2 config representation.

    The Bootstrap message is the root of the configuration. A key concept in the message is the distinction between static and dynamic resouces. Resources such as a Listener or may be supplied either statically in static_resources or have an xDS service such as or CDS configured in .

    Example

    Below we will use YAML representation of the config protos and a running example of a service proxying HTTP from 127.0.0.1:10000 to 127.0.0.2:1234.

    A minimal fully static bootstrap config is provided below:

    1. admin:
    2.   access_log_path: /tmp/admin_access.log
    3.   address:
    4.     socket_address: { address: 127.0.0.1, port_value: 9901 }
    6. static_resources:
    7.   listeners:
    8.   - name: listener_0
    9.     address:
    10.       socket_address: { address: 127.0.0.1, port_value: 10000 }
    11.     filter_chains:
    12.     - filters:
    13.       - name: envoy.http_connection_manager
    14.         config:
    15.           stat_prefix: ingress_http
    16.           codec_type: AUTO
    17.           route_config:
    18.             name: local_route
    19.             virtual_hosts:
    20.             - name: local_service
    21.               domains: ["*"]
    22.               routes:
    23.               - match: { prefix: "/" }
    24.                 route: { cluster: some_service }
    25.           http_filters:
    26.           - name: envoy.router
    27.   clusters:
    28.   - name: some_service
    29.     connect_timeout: 0.25s
    30.     type: STATIC
    31.     lb_policy: ROUND_ROBIN
    32.     load_assignment:
    33.       cluster_name: some_service
    34.       endpoints:
    35.       - lb_endpoints:
    36.         - endpoint:
    37.             address:
    38.               socket_address:
    39.                 address: 127.0.0.1
    40.                 port_value: 1234

    A bootstrap config that continues from the above example with via an EDS gRPC management server listening on 127.0.0.3:5678 is provided below:

    1. admin:
    2.   access_log_path: /tmp/admin_access.log
    3.   address:
    4.     socket_address: { address: 127.0.0.1, port_value: 9901 }
    6. static_resources:
    7.   listeners:
    8.   - name: listener_0
    9.     address:
    10.       socket_address: { address: 127.0.0.1, port_value: 10000 }
    11.     filter_chains:
    12.     - filters:
    13.       - name: envoy.http_connection_manager
    14.         config:
    15.           stat_prefix: ingress_http
    16.           codec_type: AUTO
    17.           route_config:
    18.             name: local_route
    19.             virtual_hosts:
    20.             - name: local_service
    21.               domains: ["*"]
    22.               routes:
    23.               - match: { prefix: "/" }
    24.                 route: { cluster: some_service }
    25.           - name: envoy.router
    26.   clusters:
    27.   - name: some_service
    28.     connect_timeout: 0.25s
    29.     lb_policy: ROUND_ROBIN
    30.     type: EDS
    31.     eds_cluster_config:
    32.       eds_config:
    33.         api_config_source:
    34.           api_type: GRPC
    35.           grpc_services:
    36.             envoy_grpc:
    37.               cluster_name: xds_cluster
    38.   - name: xds_cluster
    39.     connect_timeout: 0.25s
    40.     type: STATIC
    41.     lb_policy: ROUND_ROBIN
    43.     load_assignment:
    44.       cluster_name: xds_cluster
    45.       endpoints:
    46.       - lb_endpoints:
    47.         - endpoint:
    48.             address:
    49.               socket_address:
    50.                 address: 127.0.0.1
    51.                 port_value: 5678

    Notice above that xds_cluster is defined to point Envoy at the management server. Even in an otherwise completely dynamic configurations, some static resources need to be defined to point Envoy at its xDS management server(s).

    In the above example, the EDS management server could then return a proto encoding of a :

    1. version_info: "0"
    2. resources:
    3. - "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
    4.   cluster_name: some_service
    5.   endpoints:
    6.   - lb_endpoints:
    7.     - endpoint:
    8.         address:
    9.           socket_address:
    10.             address: 127.0.0.2
    11.             port_value: 1234

    The versioning and type URL scheme that appear above are explained in more detail in the streaming gRPC subscription protocol documentation.

    A fully dynamic bootstrap configuration, in which all resources other than those belonging to the management server are discovered via xDS is provided below:

    1. admin:
    2.   access_log_path: /tmp/admin_access.log
    3.   address:
    4.     socket_address: { address: 127.0.0.1, port_value: 9901 }
    6. dynamic_resources:
    7.   lds_config:
    8.     api_config_source:
    9.       api_type: GRPC
    10.       grpc_services:
    11.         envoy_grpc:
    12.           cluster_name: xds_cluster
    13.   cds_config:
    14.     api_config_source:
    15.       api_type: GRPC
    16.       grpc_services:
    17.         envoy_grpc:
    18.           cluster_name: xds_cluster
    20. static_resources:
    21.   clusters:
    22.   - name: xds_cluster
    23.     connect_timeout: 0.25s
    24.     type: STATIC
    25.     lb_policy: ROUND_ROBIN
    26.     http2_protocol_options: {}
    27.     load_assignment:
    28.       cluster_name: xds_cluster
    29.       endpoints:
    30.       - lb_endpoints:
    31.         - endpoint:
    32.             address:
    33.               socket_address:
    34.                 address: 127.0.0.1
    35.                 port_value: 5678

    The management server could respond to LDS requests with:

    1. version_info: "0"
    2. resources:
    3. - "@type": type.googleapis.com/envoy.api.v2.Listener
    4.   name: listener_0
    5.   address:
    6.     socket_address:
    7.       address: 127.0.0.1
    8.       port_value: 10000
    9.   filter_chains:
    10.     - name: envoy.http_connection_manager
    11.       config:
    12.         stat_prefix: ingress_http
    13.         codec_type: AUTO
    14.         rds:
    15.           route_config_name: local_route
    16.           config_source:
    17.             api_config_source:
    18.               api_type: GRPC
    19.               grpc_services:
    20.                 envoy_grpc:
    21.                   cluster_name: xds_cluster
    22.         http_filters:
    23.         - name: envoy.router

    The management server could respond to RDS requests with:

    1. version_info: "0"
    2. resources:
    3. - "@type": type.googleapis.com/envoy.api.v2.RouteConfiguration
    5.   virtual_hosts:
    6.   - name: local_service
    7.     domains: ["*"]
    8.     routes:
    9.     - match: { prefix: "/" }
    10.       route: { cluster: some_service }

    The management server could respond to CDS requests with:

    The management server could respond to EDS requests with:

    1. version_info: "0"
    2. resources:
    3. - "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
    4.   cluster_name: some_service
    5.   endpoints:
    6.   - lb_endpoints:
    7.     - endpoint:
    8.         address:
    9.           socket_address:
    10.             address: 127.0.0.2
    11.             port_value: 1234

    While new v2 bootstrap JSON/YAML can be written, it might be expedient to upgrade an existing v1 JSON/YAML configuration to v2. To do this (in an Envoy source tree), you can run:

    1. bazel run //tools:v1_to_bootstrap <path to v1 JSON/YAML configuration file>

    Management server

    A v2 xDS management server will implement the below endpoints as required for gRPC and/or REST serving. In both streaming gRPC and REST-JSON cases, a DiscoveryRequest is sent and a received following the xDS protocol.

    POST /envoy.api.v2.ClusterDiscoveryService/StreamClusters

    See cds.proto for the service definition. This is used by Envoy as a client when

    1. cds_config:
    2.   api_config_source:
    3.     api_type: GRPC
    4.     grpc_services:
    5.       envoy_grpc:
    6.         cluster_name: some_xds_cluster

    POST /envoy.api.v2.EndpointDiscoveryService/StreamEndpoints

    See eds.proto for the service definition. This is used by Envoy as a client when

    1. eds_config:
    2.   api_config_source:
    3.     api_type: GRPC
    4.     grpc_services:
    5.       envoy_grpc:
    6.         cluster_name: some_xds_cluster

    is set in the field of the Cluster config.

    POST /envoy.api.v2.ListenerDiscoveryService/StreamListeners

    See lds.proto for the service definition. This is used by Envoy as a client when

    1. lds_config:
    2.   api_config_source:
    3.     api_type: GRPC
    4.     grpc_services:
    5.       envoy_grpc:
    6.         cluster_name: some_xds_cluster

    is set in the of the Bootstrap config.

    POST /envoy.api.v2.RouteDiscoveryService/StreamRoutes

    See rds.proto for the service definition. This is used by Envoy as a client when

    1. route_config_name: some_route_name
    2. config_source:
    3.   api_config_source:
    4.     api_type: GRPC
    5.     grpc_services:
    6.       envoy_grpc:
    7.         cluster_name: some_xds_cluster

    is set in the field of the HttpConnectionManager config.

    POST /v2/discovery:clusters

    See cds.proto for the service definition. This is used by Envoy as a client when

    is set in the of the Bootstrap config.

    POST /v2/discovery:endpoints

    See eds.proto for the service definition. This is used by Envoy as a client when

    1. eds_config:
    2.   api_config_source:
    3.     api_type: REST
    4.     cluster_names: [some_xds_cluster]

    is set in the field of the Cluster config.

    POST /v2/discovery:listeners

    See lds.proto for the service definition. This is used by Envoy as a client when

    1. lds_config:
    2.   api_config_source:
    3.     api_type: REST
    4.     cluster_names: [some_xds_cluster]

    is set in the of the Bootstrap config.

    POST /v2/discovery:routes

    1. route_config_name: some_route_name
    2. config_source:
    3.   api_config_source:
    4.     api_type: REST
    5.     cluster_names: [some_xds_cluster]

    is set in the rds field of the config.

    While Envoy fundamentally employs an eventual consistency model, ADS provides an opportunity to sequence API update pushes and ensure affinity of a single management server for an Envoy node for API updates. ADS allows one or more APIs and their resources to be delivered on a single, bidirectional gRPC stream by the management server. Without this, some APIs such as RDS and EDS may require the management of multiple streams and connections to distinct management servers.

    ADS will allow for hitless updates of configuration by appropriate sequencing. For example, suppose foo.com was mappped to cluster X. We wish to change the mapping in the route table to point foo.com at cluster Y. In order to do this, a CDS/EDS update must first be delivered containing both clusters X and Y.

    Without ADS, the CDS/EDS/RDS streams may point at distinct management servers, or when on the same management server at distinct gRPC streams/connections that require coordination. The EDS resource requests may be split across two distinct streams, one for X and one for Y. ADS allows these to be coalesced to a single stream to a single management server, avoiding the need for distributed synchronization to correctly sequence the update. With ADS, the management server would deliver the CDS, EDS and then RDS updates on a single stream.

    ADS is only available for gRPC streaming (not REST) and is described more fully in this document. The gRPC endpoint is:

    POST /envoy.api.v2.AggregatedDiscoveryService/StreamAggregatedResources

    See discovery.proto for the service definition. This is used by Envoy as a client when

    1. ads_config:
    2.   api_type: GRPC
    3.   grpc_services:
    4.     envoy_grpc:
    5.       cluster_name: some_ads_cluster

    is set in the of the Bootstrap config.

    When this is set, any of the configuration sources can be set to use the ADS channel. For example, a LDS config could be changed from

    1. lds_config:
    2.   api_config_source:
    3.     api_type: REST

    to

    1. lds_config: {ads: {}}

    with the effect that the LDS stream will be directed to some_ads_cluster over the shared ADS channel.

    Management Server Unreachability

    When an Envoy instance loses connectivity with the management server, Envoy will latch on to the previous configuration while actively retrying in the background to reestablish the connection with the management server.

    Envoy debug logs the fact that it is not able to establish a connection with the management server every time it attempts a connection.

    statistic provides a signal for monitoring this behavior.

    Management Server has a statistics tree rooted at control_plane. with the following statistics:

    Status

    All features described in the are implemented unless otherwise noted. In the v2 API reference and the v2 API repository, all protos are frozen unless they are tagged as draft or experimental. Here, frozen means that we will not break wire format compatibility.

    Frozen protos may be further extended, e.g. by adding new fields, in a manner that does not break . Fields in the above protos may be later deprecated, subject to the breaking change policy, when their related functionality is no longer required. While frozen APIs have their wire format compatibility preserved, we reserve the right to change proto namespaces, file locations and nesting relationships, which may cause breaking code changes. We will aim to minimize the churn here.

    Protos tagged draft, meaning that they are near finalized, are likely to be at least partially implemented in Envoy but may have wire format breaking changes made prior to freezing.

    Protos tagged experimental, have the same caveats as draft protos and may have have major changes made prior to Envoy implementation and freezing.