Egress Gateways with TLS Origination (File Mount)

    • Setup Istio by following the instructions in the .

    • Start the sleep sample which will be used as a test source for external calls.

      If you have enabled , do

      Zip

      otherwise, you have to manually inject the sidecar before deploying the application:

      1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

      Note that any pod that you can exec and curl from would do.

    • Create a shell variable to hold the name of the source pod for sending requests to external services. If you used the sleep sample, run:

      1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
    • For macOS users, verify that you are using openssl version 1.1 or later:

      1. $ openssl version -a | grep OpenSSL
      2. OpenSSL 1.1.1g 21 Apr 2020

      If the previous command outputs a version 1.1 or later, as shown, your openssl command should work correctly with the instructions in this task. Otherwise, upgrade your openssl or try a different implementation of openssl, for example on a Linux machine.

    • .

    • Enable Envoy’s access logging

    This section describes how to perform the same TLS origination as in the example, only this time using an egress gateway. Note that in this case the TLS origination will be done by the egress gateway, as opposed to by the sidecar in the previous example.

    1. Define a ServiceEntry for edition.cnn.com:

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: ServiceEntry
      4. metadata:
      5. name: cnn
      6. spec:
      7. hosts:
      8. - edition.cnn.com
      9. ports:
      10. - number: 80
      11. name: http
      12. protocol: HTTP
      13. - number: 443
      14. name: https
      15. protocol: HTTPS
      16. resolution: DNS
      17. EOF
    2. Verify that your ServiceEntry was applied correctly by sending a request to http://edition.cnn.com/politics.

      1. $ kubectl exec "${SOURCE_POD}" -c sleep -- curl -sSL -o /dev/null -D - http://edition.cnn.com/politics
      2. HTTP/1.1 301 Moved Permanently
      3. ...
      4. location: https://edition.cnn.com/politics
      5. ...

      Your ServiceEntry was configured correctly if you see 301 Moved Permanently in the output.

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: istio-egressgateway
      6. spec:
      7. selector:
      8. istio: egressgateway
      9. servers:
      10. - port:
      11. number: 80
      12. name: https-port-for-tls-origination
      13. protocol: HTTPS
      14. hosts:
      15. - edition.cnn.com
      16. tls:
      17. mode: ISTIO_MUTUAL
      18. ---
      19. apiVersion: networking.istio.io/v1alpha3
      20. kind: DestinationRule
      21. metadata:
      22. name: egressgateway-for-cnn
      23. spec:
      24. host: istio-egressgateway.istio-system.svc.cluster.local
      25. subsets:
      26. - name: cnn
      27. trafficPolicy:
      28. loadBalancer:
      29. simple: ROUND_ROBIN
      30. portLevelSettings:
      31. - port:
      32. number: 80
      33. tls:
      34. mode: ISTIO_MUTUAL
      35. sni: edition.cnn.com
      36. EOF
    3. Define a VirtualService to direct the traffic through the egress gateway, and a DestinationRule to perform TLS origination for requests to edition.cnn.com:

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: VirtualService
      4. metadata:
      5. name: direct-cnn-through-egress-gateway
      6. spec:
      7. hosts:
      8. - edition.cnn.com
      9. gateways:
      10. - istio-egressgateway
      11. - mesh
      12. http:
      13. - match:
      14. - gateways:
      15. - mesh
      16. port: 80
      17. route:
      18. - destination:
      19. host: istio-egressgateway.istio-system.svc.cluster.local
      20. subset: cnn
      21. port:
      22. number: 80
      23. weight: 100
      24. - match:
      25. - gateways:
      26. - istio-egressgateway
      27. port: 80
      28. route:
      29. - destination:
      30. host: edition.cnn.com
      31. port:
      32. number: 443
      33. weight: 100
      34. ---
      35. apiVersion: networking.istio.io/v1alpha3
      36. kind: DestinationRule
      37. metadata:
      38. name: originate-tls-for-edition-cnn-com
      39. spec:
      40. host: edition.cnn.com
      41. trafficPolicy:
      42. loadBalancer:
      43. simple: ROUND_ROBIN
      44. portLevelSettings:
      45. - port:
      46. number: 443
      47. tls:
      48. mode: SIMPLE # initiates HTTPS for connections to edition.cnn.com
      49. EOF
    4. Send an HTTP request to .

      1. $ kubectl exec "${SOURCE_POD}" -c sleep -- curl -sSL -o /dev/null -D - http://edition.cnn.com/politics
      2. HTTP/1.1 200 OK
      3. ...

      The output should be the same as in the TLS Origination for Egress Traffic example, with TLS origination: without the 301 Moved Permanently message.

    Remove the Istio configuration items you created:

    Similar to the previous section, this section describes how to configure an egress gateway to perform TLS origination for an external service, only this time using a service that requires mutual TLS.

    This example is considerably more involved because you need to first:

    1. generate client and server certificates
    2. deploy an external service that supports the mutual TLS protocol
    3. redeploy the egress gateway with the needed mutual TLS certs

    Only then can you configure the external traffic to go through the egress gateway which will perform TLS origination.

    Generate client and server certificates and keys

    For this task you can use your favorite tool to generate certificates and keys. The commands below use

    1. Create a root certificate and private key to sign the certificate for your services:

      1. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
    2. Create a certificate and a private key for my-nginx.mesh-external.svc.cluster.local:

      1. $ openssl req -out my-nginx.mesh-external.svc.cluster.local.csr -newkey rsa:2048 -nodes -keyout my-nginx.mesh-external.svc.cluster.local.key -subj "/CN=my-nginx.mesh-external.svc.cluster.local/O=some organization"
      2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in my-nginx.mesh-external.svc.cluster.local.csr -out my-nginx.mesh-external.svc.cluster.local.crt
    3. Generate client certificate and private key:

      1. $ openssl req -out client.example.com.csr -newkey rsa:2048 -nodes -keyout client.example.com.key -subj "/CN=client.example.com/O=client organization"
      2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in client.example.com.csr -out client.example.com.crt

    To simulate an actual external service that supports the mutual TLS protocol, deploy an NGINX server in your Kubernetes cluster, but running outside of the Istio service mesh, i.e., in a namespace without Istio sidecar proxy injection enabled.

    1. Create a namespace to represent services outside the Istio mesh, namely mesh-external. Note that the sidecar proxy will not be automatically injected into the pods in this namespace since the automatic sidecar injection was not on it.

      1. $ kubectl create namespace mesh-external
    2. Create Kubernetes Secrets to hold the server’s and CA certificates.

      1. $ kubectl create -n mesh-external secret tls nginx-server-certs --key my-nginx.mesh-external.svc.cluster.local.key --cert my-nginx.mesh-external.svc.cluster.local.crt
      2. $ kubectl create -n mesh-external secret generic nginx-ca-certs --from-file=example.com.crt
      1. $ cat <<\EOF > ./nginx.conf
      2. events {
      3. }
      4. http {
      5. log_format main '$remote_addr - $remote_user [$time_local] $status '
      6. '"$request" $body_bytes_sent "$http_referer" '
      7. '"$http_user_agent" "$http_x_forwarded_for"';
      8. access_log /var/log/nginx/access.log main;
      9. error_log /var/log/nginx/error.log;
      10. server {
      11. listen 443 ssl;
      12. root /usr/share/nginx/html;
      13. index index.html;
      14. server_name my-nginx.mesh-external.svc.cluster.local;
      15. ssl_certificate /etc/nginx-server-certs/tls.crt;
      16. ssl_certificate_key /etc/nginx-server-certs/tls.key;
      17. ssl_client_certificate /etc/nginx-ca-certs/example.com.crt;
      18. ssl_verify_client on;
      19. }
      20. }
      21. EOF
    3. Create a Kubernetes to hold the configuration of the NGINX server:

      1. $ kubectl create configmap nginx-configmap -n mesh-external --from-file=nginx.conf=./nginx.conf
    4. Deploy the NGINX server:

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: v1
      3. kind: Service
      4. metadata:
      5. name: my-nginx
      6. namespace: mesh-external
      7. labels:
      8. run: my-nginx
      9. spec:
      10. ports:
      11. - port: 443
      12. protocol: TCP
      13. selector:
      14. run: my-nginx
      15. ---
      16. apiVersion: apps/v1
      17. kind: Deployment
      18. metadata:
      19. name: my-nginx
      20. namespace: mesh-external
      21. spec:
      22. selector:
      23. matchLabels:
      24. run: my-nginx
      25. replicas: 1
      26. template:
      27. metadata:
      28. labels:
      29. run: my-nginx
      30. spec:
      31. containers:
      32. - name: my-nginx
      33. image: nginx
      34. ports:
      35. - containerPort: 443
      36. volumeMounts:
      37. - name: nginx-config
      38. mountPath: /etc/nginx
      39. readOnly: true
      40. - name: nginx-server-certs
      41. mountPath: /etc/nginx-server-certs
      42. readOnly: true
      43. - name: nginx-ca-certs
      44. mountPath: /etc/nginx-ca-certs
      45. readOnly: true
      46. volumes:
      47. - name: nginx-config
      48. configMap:
      49. name: nginx-configmap
      50. - name: nginx-server-certs
      51. secret:
      52. secretName: nginx-server-certs
      53. - name: nginx-ca-certs
      54. secret:
      55. secretName: nginx-ca-certs
      56. EOF

    Redeploy the egress gateway with the client certificates

    1. Create Kubernetes Secrets to hold the client’s and CA certificates.

      1. $ kubectl create -n istio-system secret tls nginx-client-certs --key client.example.com.key --cert client.example.com.crt
      2. $ kubectl create -n istio-system secret generic nginx-ca-certs --from-file=example.com.crt
    2. To include a volume mounted from the new created secret, update the istio-egressgateway deployment. To patch the istio-egressgateway deployment, create the following gateway-patch.json file:

      1. $ cat > gateway-patch.json <<EOF
      2. [{
      3. "op": "add",
      4. "path": "/spec/template/spec/containers/0/volumeMounts/0",
      5. "value": {
      6. "mountPath": "/etc/istio/nginx-client-certs",
      7. "name": "nginx-client-certs",
      8. "readOnly": true
      9. }
      10. },
      11. {
      12. "op": "add",
      13. "path": "/spec/template/spec/volumes/0",
      14. "value": {
      15. "name": "nginx-client-certs",
      16. "secret": {
      17. "secretName": "nginx-client-certs",
      18. "optional": true
      19. }
      20. }
      21. },
      22. {
      23. "op": "add",
      24. "path": "/spec/template/spec/containers/0/volumeMounts/1",
      25. "value": {
      26. "mountPath": "/etc/istio/nginx-ca-certs",
      27. "name": "nginx-ca-certs",
      28. "readOnly": true
      29. },
      30. {
      31. "op": "add",
      32. "path": "/spec/template/spec/volumes/1",
      33. "value": {
      34. "name": "nginx-ca-certs",
      35. "secretName": "nginx-ca-certs",
      36. "optional": true
      37. }
      38. }
      39. }]
      40. EOF
    3. Apply istio-egressgateway deployment patch with the following command:

    4. Verify that the key and the certificate are successfully loaded in the istio-egressgateway pod:

      1. $ kubectl exec -n istio-system "$(kubectl -n istio-system get pods -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}')" -- ls -al /etc/istio/nginx-client-certs /etc/istio/nginx-ca-certs

      tls.crt and tls.key should exist in /etc/istio/nginx-client-certs, while ca-chain.cert.pem in /etc/istio/nginx-ca-certs.

    1. Create an egress Gateway for my-nginx.mesh-external.svc.cluster.local, port 443, and destination rules and virtual services to direct the traffic through the egress gateway and from the egress gateway to the external service.

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: istio-egressgateway
      6. spec:
      7. selector:
      8. istio: egressgateway
      9. servers:
      10. - port:
      11. number: 443
      12. name: https
      13. protocol: HTTPS
      14. hosts:
      15. - my-nginx.mesh-external.svc.cluster.local
      16. tls:
      17. mode: ISTIO_MUTUAL
      18. ---
      19. apiVersion: networking.istio.io/v1alpha3
      20. kind: DestinationRule
      21. metadata:
      22. name: egressgateway-for-nginx
      23. spec:
      24. host: istio-egressgateway.istio-system.svc.cluster.local
      25. subsets:
      26. - name: nginx
      27. trafficPolicy:
      28. loadBalancer:
      29. simple: ROUND_ROBIN
      30. portLevelSettings:
      31. - port:
      32. number: 443
      33. tls:
      34. mode: ISTIO_MUTUAL
      35. sni: my-nginx.mesh-external.svc.cluster.local
      36. EOF
    2. Define a VirtualService to direct the traffic through the egress gateway:

      1. $ kubectl apply -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: VirtualService
      4. metadata:
      5. name: direct-nginx-through-egress-gateway
      6. spec:
      7. hosts:
      8. - my-nginx.mesh-external.svc.cluster.local
      9. gateways:
      10. - istio-egressgateway
      11. - mesh
      12. http:
      13. - match:
      14. - gateways:
      15. - mesh
      16. port: 80
      17. route:
      18. - destination:
      19. host: istio-egressgateway.istio-system.svc.cluster.local
      20. subset: nginx
      21. port:
      22. number: 443
      23. weight: 100
      24. - match:
      25. - gateways:
      26. - istio-egressgateway
      27. port: 443
      28. route:
      29. - destination:
      30. host: my-nginx.mesh-external.svc.cluster.local
      31. port:
      32. number: 443
      33. weight: 100
      34. EOF
    3. Add a DestinationRule to perform mutual TLS origination

      1. $ kubectl apply -n istio-system -f - <<EOF
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: DestinationRule
      4. metadata:
      5. name: originate-mtls-for-nginx
      6. spec:
      7. host: my-nginx.mesh-external.svc.cluster.local
      8. trafficPolicy:
      9. loadBalancer:
      10. simple: ROUND_ROBIN
      11. portLevelSettings:
      12. - port:
      13. number: 443
      14. tls:
      15. mode: MUTUAL
      16. clientCertificate: /etc/istio/nginx-client-certs/tls.crt
      17. privateKey: /etc/istio/nginx-client-certs/tls.key
      18. caCertificates: /etc/istio/nginx-ca-certs/example.com.crt
      19. sni: my-nginx.mesh-external.svc.cluster.local
      20. EOF
    4. Send an HTTP request to http://my-nginx.mesh-external.svc.cluster.local:

      1. $ kubectl exec "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -sS http://my-nginx.mesh-external.svc.cluster.local
      2. <!DOCTYPE html>
      3. <html>
      4. <head>
      5. <title>Welcome to nginx!</title>
      6. ...
    5. Check the log of the istio-egressgateway pod for a line corresponding to our request. If Istio is deployed in the istio-system namespace, the command to print the log is:

      1. $ kubectl logs -l istio=egressgateway -n istio-system | grep 'my-nginx.mesh-external.svc.cluster.local' | grep HTTP

      You should see a line similar to the following:

      1. [2018-08-19T18:20:40.096Z] "GET / HTTP/1.1" 200 - 0 612 7 5 "172.30.146.114" "curl/7.35.0" "b942b587-fac2-9756-8ec6-303561356204" "my-nginx.mesh-external.svc.cluster.local" "172.21.72.197:443"

    Cleanup the mutual TLS origination example

    1. Remove created Kubernetes resources:

      1. $ kubectl delete secret nginx-server-certs nginx-ca-certs -n mesh-external
      2. $ kubectl delete secret istio-egressgateway-certs istio-egressgateway-ca-certs nginx-client-certs nginx-ca-certs -n istio-system
      3. $ kubectl delete configmap nginx-configmap -n mesh-external
      4. $ kubectl delete service my-nginx -n mesh-external
      5. $ kubectl delete deployment my-nginx -n mesh-external
      6. $ kubectl delete namespace mesh-external
      7. $ kubectl delete gateway istio-egressgateway
      8. $ kubectl delete virtualservice direct-nginx-through-egress-gateway
      9. $ kubectl delete destinationrule -n istio-system originate-mtls-for-nginx
      10. $ kubectl delete destinationrule egressgateway-for-nginx
    2. Delete the certificates and private keys:

      1. $ rm example.com.crt example.com.key my-nginx.mesh-external.svc.cluster.local.crt my-nginx.mesh-external.svc.cluster.local.key my-nginx.mesh-external.svc.cluster.local.csr client.example.com.crt client.example.com.csr client.example.com.key
    3. Delete the generated configuration files used in this example:

      1. $ rm ./nginx.conf
      2. $ rm ./gateway-patch.json