Egress using Wildcard Hosts

    Suppose you want to enable egress traffic in Istio for the wikipedia.org sites in all languages.Each version of wikipedia.org in a particular language has its own hostname, e.g., en.wikipedia.org andde.wikipedia.org in the English and the German languages, respectively.You want to enable egress traffic by common configuration items for all the Wikipedia sites,without the need to specify every language’s site separately.

    Before you begin

    • Setup Istio by following the instructions in the .

    • Deploy the sleep sample app to use as a test source for sending requests.If you haveenabled, run the following command to deploy the sample app:

    Zip

    Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

    You can use any pod with curl installed as a test source.

    • Set the SOURCE_POD environment variable to the name of your source pod:
    1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

    The first, and simplest, way to access a set of hosts within a common domain is by configuringa simple ServiceEntry with a wildcard host and calling the services directly from the sidecar.When calling services directly (i.e., not via an egress gateway), the configuration fora wildcard host is no different than that of any other (e.g., fully qualified) host,only much more convenient when there are many hosts within the common domain.

    • Define a ServiceEntry and corresponding VirtualSevice for *.wikipedia.org:
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: wikipedia
    6. spec:
    7.   hosts:
    8.   - "*.wikipedia.org"
    9.   ports:
    10.   - number: 443
    11.     name: tls
    12.     protocol: TLS
    13. ---
    14. apiVersion: networking.istio.io/v1alpha3
    15. kind: VirtualService
    16. metadata:
    17.   name: wikipedia
    18. spec:
    19.   hosts:
    20.   - "*.wikipedia.org"
    21.   tls:
    22.   - match:
    23.     - port: 443
    24.       sni_hosts:
    25.       - "*.wikipedia.org"
    26.     route:
    27.     - destination:
    28.         host: "*.wikipedia.org"
    29.         port:
    30.           number: 443
    31. EOF
    1. $ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>
    1. $ kubectl delete serviceentry wikipedia
    2. $ kubectl delete virtualservice wikipedia

    Configure egress gateway traffic to a wildcard host

    The configuration for accessing a wildcard host via an egress gateway depends on whether or notthe set of wildcard domains are served by a single common host.This is the case for *.wikipedia.org. All of the language-specific sites are served by everyone of the wikipedia.org servers. You can route the traffic to an IP of any *.wikipedia.org site,including www.wikipedia.org, and it will any specific site.

    In the general case, where all the domain names of a wildcard are not served by a single hosting server,a more complex configuration is required.

    • Create an egress Gateway for *.wikipedia.org, a destination rule and a virtual serviceto direct the traffic through the egress gateway and from the egress gateway to the external service.
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: istio-egressgateway
    6. spec:
    7.   selector:
    8.     istio: egressgateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: tls
    13.       protocol: TLS
    14.     hosts:
    15.     - "*.wikipedia.org"
    16.     tls:
    17.       mode: PASSTHROUGH
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.   name: egressgateway-for-wikipedia
    23. spec:
    24.   host: istio-egressgateway.istio-system.svc.cluster.local
    25.   subsets:
    26.     - name: wikipedia
    27. ---
    28. apiVersion: networking.istio.io/v1alpha3
    29. kind: VirtualService
    30. metadata:
    31.   name: direct-wikipedia-through-egress-gateway
    32. spec:
    33.   hosts:
    34.   - "*.wikipedia.org"
    35.   gateways:
    36.   - mesh
    37.   - istio-egressgateway
    38.   tls:
    39.   - match:
    40.     - gateways:
    41.       - mesh
    42.       port: 443
    43.       sni_hosts:
    44.       - "*.wikipedia.org"
    45.     route:
    46.     - destination:
    47.         host: istio-egressgateway.istio-system.svc.cluster.local
    48.         subset: wikipedia
    49.         port:
    50.           number: 443
    51.       weight: 100
    52.   - match:
    53.     - gateways:
    54.       - istio-egressgateway
    55.       port: 443
    56.       sni_hosts:
    57.       - "*.wikipedia.org"
    58.     route:
    59.     - destination:
    60.         host: www.wikipedia.org
    61.           number: 443
    62.       weight: 100
    63. EOF
    • Create a ServiceEntry for the destination server, www.wikipedia.org.
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    6. spec:
    7.   hosts:
    8.   - www.wikipedia.org
    9.   ports:
    10.   - number: 443
    11.     name: tls
    12.     protocol: TLS
    13.   resolution: DNS
    14. EOF
    1. $ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>
    • Check the statistics of the egress gateway’s proxy for the counter that corresponds to yourrequests to *.wikipedia.org. If Istio is deployed in the istio-system namespace, the command to print thecounter is:

    Cleanup wildcard configuration for a single hosting server

    1. $ kubectl delete serviceentry www-wikipedia
    2. $ kubectl delete gateway istio-egressgateway
    3. $ kubectl delete virtualservice direct-wikipedia-through-egress-gateway
    4. $ kubectl delete destinationrule egressgateway-for-wikipedia

    The configuration in the previous section worked because all the *.wikipedia.org sites canbe served by any one of the wikipedia.org servers. However, this is not always the case.For example, you may want to configure egress control for access to more generalwildcard domains like .com or .org.

    Configuring traffic to arbitrary wildcard domains introduces a challenge for Istio gateways. In the previous sectionyou directed the traffic to www.wikipedia.org, which was made known to your gateway during configuration.The gateway, however, would not know the IP address of any arbitrary host it receives in a request.This is due to a limitation of , the proxy used by the default Istio egress gateway.Envoy routes traffic either to predefined hosts, predefined IP addresses, or to the original destination IP address ofthe request. In the gateway case, the original destination IP of the request is lost since the request is first routedto the egress gateway and its destination IP address is the IP address of the gateway.

    Consequently, the Istio gateway based on Envoy cannot route traffic to an arbitrary host that is not preconfigured,and therefore is unable to perform traffic control for arbitrary wildcard domains.To enable such traffic control for HTTPS, and for any TLS, you need to deploy an SNI forward proxy in addition to Envoy.Envoy will route the requests destined for a wildcard domain to the SNI forward proxy, which, in turn, will forward therequests to the destination specified by the SNI value.

    The egress gateway with SNI proxy and the related parts of the Istio architecture are shown in the following diagram:

    Egress Gateway with SNI proxy

    The following sections show you how to redeploy the egress gateway with an SNI proxy and then configure Istio to routeHTTPS traffic through the gateway to arbitrary wildcard domains.

    Setup egress gateway with SNI proxy

    In this section you deploy an egress gateway with an SNI proxy in addition to the standard Istio Envoy proxy.This example uses Nginx for the SNI proxy, although any SNI proxy that is capable of routing trafficaccording to arbitrary, not-preconfigured, SNI values would do.The SNI proxy will listen on port 8443, although you can use any port other than the ports specified forthe egress Gateway and for the VirtualServices bound to it.The SNI proxy will forward the traffic to port 443.

    • Create a configuration file for the Nginx SNI proxy. You may want to edit the file to specify additional Nginxsettings, if required. Note that the listen directive of the server specifies port 8443, its proxy_passdirective uses ssl_preread_server_name with port 443 and ssl_preread is on to enable SNI reading.
    1. $ cat <<EOF > ./sni-proxy.conf
    2. user www-data;
    4. events {
    5. }
    7. stream {
    8.   log_format log_stream '\$remote_addr [\$time_local] \$protocol [\$ssl_preread_server_name]'
    9.   '\$status \$bytes_sent \$bytes_received \$session_time';
    11.   access_log /var/log/nginx/access.log log_stream;
    12.   error_log  /var/log/nginx/error.log;
    14.   # tcp forward proxy by SNI
    15.   server {
    16.     resolver 8.8.8.8 ipv6=off;
    17.     listen       127.0.0.1:8443;
    18.     proxy_pass   \$ssl_preread_server_name:443;
    19.     ssl_preread  on;
    20.   }
    21. }
    22. EOF
    • Create a Kubernetes to hold the configuration of the Nginx SNI proxy:
    1. $ kubectl create configmap egress-sni-proxy-configmap -n istio-system --from-file=nginx.conf=./sni-proxy.conf
    • The following command will generate istio-egressgateway-with-sni-proxy.yaml which you can optionally edit and then deploy.
    1. $ cat <<EOF | istioctl manifest generate --set values.global.istioNamespace=istio-system -f - > ./istio-egressgateway-with-sni-proxy.yaml
    2. gateways:
    3.   enabled: true
    4.   istio-ingressgateway:
    5.     enabled: false
    6.   istio-egressgateway:
    7.     enabled: false
    8.   istio-egressgateway-with-sni-proxy:
    9.     enabled: true
    10.     labels:
    11.       app: istio-egressgateway-with-sni-proxy
    12.       istio: egressgateway-with-sni-proxy
    13.     replicaCount: 1
    14.     autoscaleMin: 1
    15.     autoscaleMax: 5
    16.     cpu:
    17.       targetAverageUtilization: 80
    18.     serviceAnnotations: {}
    19.     type: ClusterIP
    20.     ports:
    21.       - port: 443
    22.         name: https
    23.     secretVolumes:
    24.       - name: egressgateway-certs
    25.         secretName: istio-egressgateway-certs
    26.         mountPath: /etc/istio/egressgateway-certs
    27.       - name: egressgateway-ca-certs
    28.         secretName: istio-egressgateway-ca-certs
    29.         mountPath: /etc/istio/egressgateway-ca-certs
    30.     configVolumes:
    31.       - name: sni-proxy-config
    32.         configMapName: egress-sni-proxy-configmap
    33.     additionalContainers:
    34.     - name: sni-proxy
    35.       image: nginx
    36.       volumeMounts:
    37.       - name: sni-proxy-config
    38.         mountPath: /etc/nginx
    39.         readOnly: true
    40. EOF
    • Deploy the new egress gateway:
    1. $ kubectl apply -f ./istio-egressgateway-with-sni-proxy.yaml
    2. serviceaccount "istio-egressgateway-with-sni-proxy-service-account" created
    3. role "istio-egressgateway-with-sni-proxy-istio-system" created
    4. rolebinding "istio-egressgateway-with-sni-proxy-istio-system" created
    5. service "istio-egressgateway-with-sni-proxy" created
    6. deployment "istio-egressgateway-with-sni-proxy" created
    7. horizontalpodautoscaler "istio-egressgateway-with-sni-proxy" created
    • Verify that the new egress gateway is running. Note that the pod has two containers (one is the Envoy proxy and thesecond one is the SNI proxy).
    1. $ kubectl get pod -l istio=egressgateway-with-sni-proxy -n istio-system
    2. NAME                                                  READY     STATUS    RESTARTS   AGE
    3. istio-egressgateway-with-sni-proxy-79f6744569-pf9t2   2/2       Running   0          17s
    • Create a service entry with a static address equal to 127.0.0.1 (localhost), and disable mutual TLS for traffic directed to the newservice entry:
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: sni-proxy
    6. spec:
    7.   hosts:
    8.   - sni-proxy.local
    9.   location: MESH_EXTERNAL
    10.   ports:
    11.   - number: 8443
    12.     name: tcp
    14.   endpoints:
    15.   - address: 127.0.0.1
    16. ---
    17. apiVersion: networking.istio.io/v1alpha3
    18. kind: DestinationRule
    19. metadata:
    20.   name: disable-mtls-for-sni-proxy
    21. spec:
    22.   host: sni-proxy.local
    23.   trafficPolicy:
    24.     tls:
    25.       mode: DISABLE
    26. EOF

    Configure traffic through egress gateway with SNI proxy

    • Define a ServiceEntry for *.wikipedia.org:
    1. $ cat <<EOF | kubectl create -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: wikipedia
    6. spec:
    7.   hosts:
    8.   - "*.wikipedia.org"
    9.   ports:
    10.   - number: 443
    11.     name: tls
    12.     protocol: TLS
    13. EOF
    • Create an egress Gateway for *.wikipedia.org, port 443, protocol TLS, and a virtual service to direct thetraffic destined for *.wikipedia.org through the gateway.

    Choose the instructions corresponding to whether or not you want to enable between the source pod and the egress gateway.

    You may want to enable mutual TLS to let the egress gateway monitor the identity of the source pods and to enable Mixer policy enforcement based on that identity.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.  name: istio-egressgateway-with-sni-proxy
    6. spec:
    7.  selector:
    8.    istio: egressgateway-with-sni-proxy
    9.  servers:
    10.  - port:
    11.      number: 443
    12.      name: tls
    13.      protocol: TLS
    14.    hosts:
    15.    - "*.wikipedia.org"
    16.    tls:
    17.      mode: PASSTHROUGH
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.  name: egressgateway-for-wikipedia
    23. spec:
    24.  host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
    25.  subsets:
    26.    - name: wikipedia
    27. ---
    28. apiVersion: networking.istio.io/v1alpha3
    29. kind: VirtualService
    30. metadata:
    31.  name: direct-wikipedia-through-egress-gateway
    32. spec:
    33.  hosts:
    34.  - "*.wikipedia.org"
    35.  gateways:
    36.  - mesh
    37.  - istio-egressgateway-with-sni-proxy
    38.  tls:
    39.  - match:
    40.    - gateways:
    41.      - mesh
    42.      port: 443
    43.      sni_hosts:
    44.      - "*.wikipedia.org"
    45.    route:
    46.    - destination:
    47.        host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
    48.        subset: wikipedia
    49.        port:
    50.          number: 443
    51.      weight: 100
    52.  - match:
    53.    - gateways:
    54.      - istio-egressgateway-with-sni-proxy
    55.      port: 443
    56.      sni_hosts:
    57.      - "*.wikipedia.org"
    58.    route:
    59.    - destination:
    60.        host: sni-proxy.local
    61.        port:
    62.          number: 8443
    63.      weight: 100
    64. EOF
    1. $ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>
    • Check the log of the egress gateway’s Envoy proxy. If Istio is deployed in the istio-system namespace, the command toprint the log is:
    1. $ kubectl logs -l istio=egressgateway-with-sni-proxy -c istio-proxy -n istio-system

    You should see lines similar to the following:

    1. [2019-01-02T16:34:23.312Z] "- - -" 0 - 578 79141 624 - "-" "-" "-" "-" "127.0.0.1:8443" outbound|8443||sni-proxy.local 127.0.0.1:55018 172.30.109.84:443 172.30.109.112:45346 en.wikipedia.org
    2. [2019-01-02T16:34:24.079Z] "- - -" 0 - 586 65770 638 - "-" "-" "-" "-" "127.0.0.1:8443" outbound|8443||sni-proxy.local 127.0.0.1:55034 172.30.109.84:443 172.30.109.112:45362 de.wikipedia.org
    • Check the logs of the SNI proxy. If Istio is deployed in the istio-system namespace, the command to print thelog is:
    1. $ kubectl logs -l istio=egressgateway-with-sni-proxy -n istio-system -c sni-proxy
    2. 127.0.0.1 [01/Aug/2018:15:32:02 +0000] TCP [en.wikipedia.org]200 81513 280 0.600
    3. 127.0.0.1 [01/Aug/2018:15:32:03 +0000] TCP [de.wikipedia.org]200 67745 291 0.659
    • Check the mixer log. If Istio is deployed in the istio-system namespace, the command to print thelog is:
    1. $ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep '"connectionEvent":"open"' | grep '"sourceName":"istio-egressgateway' | grep 'wikipedia.org'
    2. {"level":"info","time":"2018-08-26T16:16:34.784571Z","instance":"tcpaccesslog.logentry.istio-system","connectionDuration":"0s","connectionEvent":"open","connection_security_policy":"unknown","destinationApp":"","destinationIp":"127.0.0.1","destinationName":"unknown","destinationNamespace":"default","destinationOwner":"unknown","destinationPrincipal":"cluster.local/ns/istio-system/sa/istio-egressgateway-with-sni-proxy-service-account","destinationServiceHost":"","destinationWorkload":"unknown","protocol":"tcp","receivedBytes":298,"reporter":"source","requestedServerName":"en.wikipedia.org","sentBytes":0,"sourceApp":"istio-egressgateway-with-sni-proxy","sourceIp":"172.30.146.88","sourceName":"istio-egressgateway-with-sni-proxy-7c4f7868fb-rc8pr","sourceNamespace":"istio-system","sourceOwner":"kubernetes://apis/extensions/v1beta1/namespaces/istio-system/deployments/istio-egressgateway-with-sni-proxy","sourcePrincipal":"cluster.local/ns/sleep/sa/default","sourceWorkload":"istio-egressgateway-with-sni-proxy","totalReceivedBytes":298,"totalSentBytes":0}

    Note the requestedServerName attribute.

    Cleanup wildcard configuration for arbitrary domains

    • Delete the configuration items for *.wikipedia.org:
    1. $ kubectl delete serviceentry wikipedia
    2. $ kubectl delete gateway istio-egressgateway-with-sni-proxy
    3. $ kubectl delete virtualservice direct-wikipedia-through-egress-gateway
    4. $ kubectl delete destinationrule egressgateway-for-wikipedia
    5. $ kubectl delete --ignore-not-found=true envoyfilter forward-downstream-sni egress-gateway-sni-verifier
    • Delete the configuration items for the egressgateway-with-sni-proxy Deployment:
    1. $ kubectl delete serviceentry sni-proxy
    2. $ kubectl delete destinationrule disable-mtls-for-sni-proxy
    3. $ kubectl delete -f ./istio-egressgateway-with-sni-proxy.yaml
    4. $ kubectl delete configmap egress-sni-proxy-configmap -n istio-system
    • Remove the configuration files you created:

      See also

      Comparison of alternative solutions to control egress traffic including performance considerations.

      Secure Control of Egress Traffic in Istio, part 2

      Use Istio Egress Traffic Control to prevent attacks involving egress traffic.

      Attacks involving egress traffic and requirements for egress traffic control.

      Egress Gateway Performance Investigation

      Verifies the performance impact of adding an egress gateway.

      Describes a simple scenario based on Istio's Bookinfo example.

      Monitoring and Access Policies for HTTP Egress Traffic