Setting up Cilium in AWS ENI mode

    The AWS ENI integration is still subject to some limitations. See Limitations for details.

    Setup a Kubernetes on AWS. You can use any method you prefer, but for the simplicity of this tutorial, we are going to use . For more details on how to set up an EKS cluster using eksctl, see the section Installation on AWS EKS.

    Disable VPC CNI (aws-node DaemonSet) (EKS only)

    If you are running an EKS cluster, you should delete the aws-node DaemonSet.

    Cilium will manage ENIs instead of VPC CNI, so the aws-node DaemonSet has to be deleted to prevent conflict behavior.

    Note

    Once aws-node DaemonSet is deleted, EKS will not try to restore it.

    1. kubectl -n kube-system delete daemonset aws-node

    Deploy Cilium

    Note

    First, make sure you have Helm 3 .

    If you have (or planning to have) Helm 2 charts (and Tiller) in the same cluster, there should be no issue as both version are mutually compatible in order to support gradual migration. Cilium chart is targeting Helm 3 (v3.0.3 and above).

    Setup Helm repository:

    Deploy Cilium release via Helm:

    1. helm install cilium cilium/cilium --version 1.8.10 \
    2.   --namespace kube-system \
    3.   --set global.eni=true \
    4.   --set config.ipam=eni \
    5.   --set global.egressMasqueradeInterfaces=eth0 \
    6.   --set global.tunnel=disabled \
    7.   --set global.nodeinit.enabled=true

    Note

    The above options are assuming that masquerading is desired and that the VM is connected to the VPC using eth0. It will route all traffic that does not stay in the VPC via eth0 and masquerade it.

    If you want to avoid masquerading, set global.masquerade=false. You must ensure that the security groups associated with the ENIs (eth1, eth2, …) allow for egress traffic to outside of the VPC. By default, the security groups for pod ENIs are derived from the primary ENI (eth0).

    1. eksctl create nodegroup --cluster test-cluster --nodes 2

    Validate the Installation

    You can monitor as Cilium and all required components are being installed:

    1. kubectl -n kube-system get pods --watch
    2. NAME                                    READY   STATUS              RESTARTS   AGE
    3. cilium-operator-cb4578bc5-q52qk         0/1     Pending             0          8s
    4. cilium-s8w5m                            0/1     PodInitializing     0          7s
    5. coredns-86c58d9df4-4g7dd                0/1     ContainerCreating   0          8m57s
    6. coredns-86c58d9df4-4l6b2                0/1     ContainerCreating   0          8m57s

    It may take a couple of minutes for all components to come up:

    1. cilium-operator-cb4578bc5-q52qk         1/1     Running   0          4m13s
    2. cilium-s8w5m                            1/1     Running   0          4m12s
    3. coredns-86c58d9df4-4g7dd                1/1     Running   0          13m
    4. coredns-86c58d9df4-4l6b2                1/1     Running   0          13m

    Deploy the check with:

      It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

      1. $ kubectl get pods -n cilium-test
      2. NAME                                                    READY   STATUS    RESTARTS   AGE
      4. echo-b-59757679d4-pjtdl                                 1/1     Running   0          69s
      5. echo-b-host-f86bd784d-wnh4v                             1/1     Running   0          68s
      6. host-to-b-multi-node-clusterip-585db65b4d-x74nz         1/1     Running   0          68s
      7. host-to-b-multi-node-headless-77c64bc7d8-kgf8p          1/1     Running   0          67s
      8. pod-to-a-allowed-cnp-87b5895c8-bfw4x                    1/1     Running   0          68s
      9. pod-to-a-b76ddb6b4-2v4kb                                1/1     Running   0          68s
      10. pod-to-a-denied-cnp-677d9f567b-kkjp4                    1/1     Running   0          68s
      11. pod-to-b-intra-node-nodeport-8484fb6d89-bwj8q           1/1     Running   0          68s
      12. pod-to-b-multi-node-clusterip-f7655dbc8-h5bwk           1/1     Running   0          68s
      13. pod-to-b-multi-node-headless-5fd98b9648-5bjj8           1/1     Running   0          68s
      14. pod-to-b-multi-node-nodeport-74bd8d7bd5-kmfmm           1/1     Running   0          68s
      15. pod-to-external-1111-7489c7c46d-jhtkr                   1/1     Running   0          68s
      16. pod-to-external-fqdn-allow-google-cnp-b7b6bcdcb-97p75   1/1     Running   0          68s

      Note

      If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

      Specify Environment Variables

      Specify the namespace in which Cilium is installed as CILIUM_NAMESPACE environment variable. Subsequent commands reference this environment variable.

      1. export CILIUM_NAMESPACE=kube-system

      Hubble is a fully distributed networking and security observability platform for cloud native workloads. It is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner.

      • Hubble can be configured to be in local mode or distributed mode (beta).

        Local Mode

        Distributed Mode (beta)

        In local mode, Hubble listens on a UNIX domain socket. You can connect to a Hubble instance by running hubble command from inside the Cilium pod. This provides networking visibility for traffic observed by the local Cilium agent.

        1. helm upgrade cilium cilium/cilium --version 1.8.10 \
        2.    --namespace $CILIUM_NAMESPACE \
        3.    --reuse-values \
        4.    --set global.hubble.enabled=true \
        5.    --set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

        In distributed mode (beta), Hubble listens on a TCP port on the host network. This allows to communicate with all the Hubble instances in the cluster. Hubble CLI and Hubble UI in turn connect to Hubble Relay to provide cluster-wide networking visibility.

        Warning

        In Distributed mode, Hubble runs a gRPC service over plain-text HTTP on the host network without any authentication/authorization. The main consequence is that anybody who can reach the Hubble gRPC service can obtain all the networking metadata from the host. It is therefore strongly discouraged to enable distributed mode in a production environment.

        1. helm upgrade cilium cilium/cilium --version 1.8.10 \
        2.    --namespace $CILIUM_NAMESPACE \
        3.    --reuse-values \
        4.    --set global.hubble.enabled=true \
        5.    --set global.hubble.listenAddress=":4244" \
        6.    --set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
        7.    --set global.hubble.relay.enabled=true \
        8.    --set global.hubble.ui.enabled=true

      • To pick one Cilium instance and validate that Hubble is properly configured to listen on a UNIX domain socket:

      • (Distributed mode only) To validate that Hubble Relay is running, install the hubble CLI:

        MacOS

        Windows

        Download the latest hubble release:

        1. export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
        2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
        4. sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
        5. tar zxf hubble-linux-amd64.tar.gz

        and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

        1. sudo mv hubble /usr/local/bin

        Download the latest hubble release:

        1. export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
        2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz"
        3. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz.sha256sum"
        4. shasum -a 256 -c hubble-darwin-amd64.tar.gz.sha256sum
        5. tar zxf hubble-darwin-amd64.tar.gz

        and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

        1. sudo mv hubble /usr/local/bin

        Download the latest hubble release:

        1. curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
        2. set /p HUBBLE_VERSION=<stable.txt
        3. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
        4. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
        5. certutil -hashfile hubble-windows-amd64.tar.gz SHA256
        6. type hubble-windows-amd64.tar.gz.sha256sum
        7. :: verify that the checksum from the two commands above match
        8. tar zxf hubble-windows-amd64.tar.gz

        and move the hubble.exe CLI to a directory listed in the %PATH% environment variable after extracting it from the tarball.

        Once the hubble CLI is installed, set up a port forwarding for hubble-relay service and run hubble observe command:

        1. kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay 4245:80
        2. hubble observe --server localhost:4245

        (For Linux / MacOS) For convenience, you may set and export the HUBBLE_DEFAULT_SOCKET_PATH environment variable:

        This will allow you to use hubble status and hubble observe commands without having to specify the server address via the --server flag.

      • (Distributed mode only) To validate that Hubble UI is properly configured, set up a port forwarding for hubble-ui service:

        1. kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-ui 12000:80

        and then open http://localhost:12000/.

      Limitations

      • The AWS ENI integration of Cilium is currently only enabled for IPv4.

      • When applying L7 policies at egress, the source identity context is lost as it is currently not carried in the packet. This means that traffic will look like it is coming from outside of the cluster to the receiving pod.

      • HostPort type services additionally require either of the following configurations:

      Troubleshooting

      Make sure to disable DHCP on ENIs