RKE Hardening Guide

    备注

    This hardening guide describes how to secure the nodes in your cluster. We recommended that you follow this guide before you install Kubernetes.

    This hardening guide is intended to be used for RKE clusters and is associated with the following versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:

    RKE Hardening Guides - 图2备注

    At the time of writing, the upstream CIS Kubernetes v1.25 benchmark is not yet available in Rancher. At this time Rancher is using the CIS v1.23 benchmark when scanning Kubernetes v1.25 clusters. Due to that, the CIS checks 5.2.3, 5.2.4, 5.2.5 and 5.2.6 might fail.

    For more details on how to evaluate a hardened RKE cluster against the official CIS benchmark, refer to the RKE self-assessment guides for specific Kubernetes and CIS benchmark versions.

    The following sysctl configuration is recommended for all nodes types in the cluster. Set the following parameters in /etc/sysctl.d/90-kubelet.conf:

    Run sysctl -p /etc/sysctl.d/90-kubelet.conf to enable the settings.

    A user account and group for the etcd service is required to be set up before installing RKE.

    Create etcd user and group

    1. groupadd --gid 52034 etcd
    2. useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin

    When deploying RKE through its cluster configuration config.yml file, update the uid and gid of the etcd user:

    1. services:
    2.   etcd:
    3.     gid: 52034
    4.     uid: 52034

    Kubernetes runtime requirements

    Set automountServiceAccountToken to false for default service accounts

    Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.

    For each namespace including default and kube-system on a standard RKE install, the default service account must include this value:

    1. automountServiceAccountToken: false

    Save the following configuration to a file called account_update.yaml.

    Create a bash script file called account_update.sh. Be sure to chmod +x account_update.sh so the script has execute permissions.

    1. #!/bin/bash -e
    3. for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
    4.   kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
    5. done

    Execute this script to apply the account_update.yaml configuration to default service account in all namespaces.

    Ensure that all Namespaces have Network Policies defined

    Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints.

    Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace. To enforce network policies, a container network interface (CNI) plugin must be enabled. This guide uses to provide the policy enforcement. Additional information about CNI providers can be found here.

    Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a permissive example is provided below. If you want to allow all traffic to all pods in a namespace (even if policies are added that cause some pods to be treated as “isolated”), you can create a policy that explicitly allows all traffic in that namespace. Save the following configuration as default-allow-all.yaml. Additional about network policies can be found on the Kubernetes site.

    This network policy is just an example and is not recommended for production use.

    1. ---
    2. apiVersion: networking.k8s.io/v1
    3. kind: NetworkPolicy
    4. metadata:
    5.   name: default-allow-all
    6. spec:
    7.   podSelector: {}
    8.   ingress:
    9.   - {}
    10.   egress:
    11.   - {}
    12.   policyTypes:
    13.   - Ingress
    14.   - Egress

    Create a bash script file called apply_networkPolicy_to_all_ns.sh. Be sure to chmod +x apply_networkPolicy_to_all_ns.sh so the script has execute permissions.

    1. #!/bin/bash -e
    3. for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
    4.   kubectl apply -f default-allow-all.yaml -n ${namespace}
    5. done

    Execute this script to apply the default-allow-all.yaml configuration with the permissive NetworkPolicy to all namespaces.

    • Rancher exec shell and view logs for pods are not functional in a hardened setup when only a public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
    • When setting default_pod_security_policy_template_id: to restricted or restricted-noroot, based on the pod security policies (PSP) by Rancher, Rancher creates RoleBindings and ClusterRoleBindings on the default service accounts. The CIS check 5.1.5 requires that the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition, the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.

    Reference Hardened RKE cluster.yml Configuration

    The reference cluster.yml is used by the RKE CLI that provides the configuration needed to achieve a hardened installation of RKE. RKE provides additional details about the configuration items. This reference cluster.yml does not include the required nodes directive which will vary depending on your environment. Documentation for node configuration in RKE can be found here.

    The example cluster.yml configuration file contains an Admission Configuration policy in the services.kube-api.admission_configuration field. This policy contains the namespace exemptions necessary for an imported RKE cluster to run properly in Rancher, similar to Rancher’s pre-defined rancher-restricted policy.

    If you prefer to use RKE’s default restricted policy, then leave the services.kube-api.admission_configuration field empty and set services.pod_security_configuration to restricted. See for more information.

    • v1.25 and Newer
    • v1.24 and Older

    备注

    If you intend to import an RKE cluster into Rancher, please consult the documentation for how to configure the PSA to exempt Rancher system namespaces.

    1. # If you intend to deploy Kubernetes in an air-gapped environment,
    2. # please consult the documentation on how to configure custom RKE images.
    3. nodes: []
    4. kubernetes_version: # Define RKE version
    5. services:
    6.   etcd:
    7.     uid: 52034
    8.     gid: 52034
    9.   kube-api:
    10.     secrets_encryption_config:
    11.       enabled: true
    12.     audit_log:
    14.     event_rate_limit:
    15.       enabled: true
    16.     pod_security_policy: true
    17.   kube-controller:
    18.     extra_args:
    19.       feature-gates: RotateKubeletServerCertificate=true
    20.   kubelet:
    21.     extra_args:
    22.       feature-gates: RotateKubeletServerCertificate=true
    23.       protect-kernel-defaults: true
    24.     generate_serving_certificate: true
    25. addons: |
    26.   # Upstream Kubernetes restricted PSP policy
    27.   # https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml
    28.   apiVersion: policy/v1beta1
    29.   kind: PodSecurityPolicy
    30.   metadata:
    31.     name: restricted-noroot
    32.   spec:
    33.     privileged: false
    34.     # Required to prevent escalations to root.
    35.     allowPrivilegeEscalation: false
    36.     requiredDropCapabilities:
    37.     # Allow core volume types.
    38.     volumes:
    39.       - 'configMap'
    40.       - 'emptyDir'
    41.       - 'projected'
    42.       - 'secret'
    43.       - 'downwardAPI'
    44.       # Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use.
    45.       - 'csi'
    46.       - 'persistentVolumeClaim'
    47.       - 'ephemeral'
    48.     hostNetwork: false
    49.     hostIPC: false
    50.     hostPID: false
    51.     runAsUser:
    52.       # Require the container to run without root privileges.
    53.       rule: 'MustRunAsNonRoot'
    54.     seLinux:
    55.       # This policy assumes the nodes are using AppArmor rather than SELinux.
    56.       rule: 'RunAsAny'
    57.     supplementalGroups:
    58.       rule: 'MustRunAs'
    59.       ranges:
    60.         # Forbid adding the root group.
    61.         - min: 1
    62.           max: 65535
    63.     fsGroup:
    64.       rule: 'MustRunAs'
    65.       ranges:
    66.         # Forbid adding the root group.
    67.         - min: 1
    68.           max: 65535
    69.     readOnlyRootFilesystem: false
    70.   ---
    71.   apiVersion: rbac.authorization.k8s.io/v1
    72.   kind: ClusterRole
    73.   metadata:
    74.     name: psp:restricted-noroot
    75.   rules:
    76.   - apiGroups:
    77.     - extensions
    78.     resourceNames:
    79.     - restricted-noroot
    80.     resources:
    81.     - podsecuritypolicies
    82.     verbs:
    83.     - use
    84.   ---
    85.   apiVersion: rbac.authorization.k8s.io/v1
    86.   kind: ClusterRoleBinding
    87.   metadata:
    88.     name: psp:restricted-noroot
    89.   roleRef:
    90.     apiGroup: rbac.authorization.k8s.io
    91.     kind: ClusterRole
    92.     name: psp:restricted-noroot
    93.   subjects:
    94.   - apiGroup: rbac.authorization.k8s.io
    95.     kind: Group
    96.     name: system:serviceaccounts
    97.   - apiGroup: rbac.authorization.k8s.io
    98.     kind: Group
    99.     name: system:authenticated
    100.   ---
    101.   apiVersion: networking.k8s.io/v1
    102.   kind: NetworkPolicy
    103.   metadata:
    104.     name: default-allow-all
    105.   spec:
    106.     podSelector: {}
    107.     ingress:
    109.     egress:
    110.     - {}
    111.     policyTypes:
    112.     - Ingress
    113.     - Egress
    114.   ---
    115.   apiVersion: v1
    116.   kind: ServiceAccount
    117.   metadata:
    118.     name: default
    119.   automountServiceAccountToken: false

    The reference RKE cluster template provides the minimum required configuration to achieve a hardened installation of Kubernetes. RKE templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher documentation for additional information about installing RKE and its template details.

    • v1.25 and Newer
    1. #
    2. # Cluster Config
    3. #
    4. default_pod_security_admission_configuration_template_name: rancher-restricted
    5. enable_network_policy: true
    6. local_cluster_auth_endpoint:
    7.   enabled: true
    8. name: # Define cluster name
    10. #
    11. # Rancher Config
    12. #
    13. rancher_kubernetes_engine_config:
    14.   addon_job_timeout: 45
    15.   authentication:
    16.     strategy: x509|webhook
    17.   kubernetes_version: # Define RKE version
    18.   services:
    19.     etcd:
    20.       uid: 52034
    21.       gid: 52034
    22.     kube-api:
    23.       audit_log:
    24.         enabled: true
    25.       event_rate_limit:
    26.         enabled: true
    27.       pod_security_policy: false
    28.       secrets_encryption_config:
    29.         enabled: true
    30.     kube-controller:
    31.       extra_args:
    32.         feature-gates: RotateKubeletServerCertificate=true
    33.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
    34.     kubelet:
    35.       extra_args:
    36.         feature-gates: RotateKubeletServerCertificate=true
    37.         protect-kernel-defaults: true
    38.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
    39.       generate_serving_certificate: true
    40.     scheduler:
    41.       extra_args:
    42.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
    1. #
    2. # Cluster Config
    3. #
    4. default_pod_security_policy_template_id: restricted-noroot
    5. enable_network_policy: true
    6. local_cluster_auth_endpoint:
    7.   enabled: true
    8. name: # Define cluster name
    10. #
    11. # Rancher Config
    12. #
    13. rancher_kubernetes_engine_config:
    14.   addon_job_timeout: 45
    15.   authentication:
    16.     strategy: x509|webhook
    17.   kubernetes_version: # Define RKE version
    18.   services:
    19.     etcd:
    20.       uid: 52034
    21.       gid: 52034
    22.     kube-api:
    23.       audit_log:
    24.         enabled: true
    25.       event_rate_limit:
    26.         enabled: true
    27.       pod_security_policy: true
    28.       secrets_encryption_config:
    29.         enabled: true
    30.     kube-controller:
    31.       extra_args:
    32.         feature-gates: RotateKubeletServerCertificate=true
    33.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
    34.     kubelet:
    35.       extra_args:
    36.         feature-gates: RotateKubeletServerCertificate=true
    37.         protect-kernel-defaults: true
    38.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
    39.       generate_serving_certificate: true
    40.     scheduler:
    41.       extra_args:
    42.         tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256

    Conclusion