kubeadm

    以下是详细的 kubeadm 部署集群步骤。

    所有机器都需要初始化 docker 和 kubelet。

    2. apt-get update
    3. apt-get install -y apt-transport-https ca-certificates curl software-properties-common
    4. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    5. add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
    6. apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
    8. apt-get update && apt-get install -y apt-transport-https curl
    9. curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
    10.     cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    11. deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
    12. EOF
    13. apt-get update
    14. apt-get install -y kubelet kubeadm kubectl

    CentOS

    1. yum install -y docker
    2. systemctl enable docker && systemctl start docker
    4. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    5. [kubernetes]
    6. name=Kubernetes
    7. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    8. enabled=1
    9. gpgcheck=1
    10. repo_gpgcheck=1
    11. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    12. EOF
    13. setenforce 0
    14. sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
    15. yum install -y kubelet kubeadm kubectl
    16. systemctl enable kubelet && systemctl start kubelet

    安装 master

    1. # --api-advertise-addresses <ip-address>
    2. # for flannel, setup --pod-network-cidr 10.244.0.0/16
    3. kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version latest
    5. # enable schedule pods on the master
    6. export KUBECONFIG=/etc/kubernetes/admin.conf
    7. kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-

    如果需要修改 kubernetes 服务的配置选项，则需要创建一个 kubeadm 配置文件，其格式为

    1. apiVersion: kubeadm.k8s.io/v1alpha3
    2. kind: InitConfiguration
    3. bootstrapTokens:
    4. - token: "9a08jv.c0izixklcxtmnze7"
    5.   description: "kubeadm bootstrap token"
    6.   ttl: "24h"
    7. - token: "783bde.3f89s0fje9f38fhf"
    8.   description: "another bootstrap token"
    9.   usages:
    10.   - signing
    11.   groups:
    12.   - system:anonymous
    13. nodeRegistration:
    14.   name: "ec2-10-100-0-1"
    15.   criSocket: "/var/run/dockershim.sock"
    16.   taints:
    17.   - key: "kubeadmNode"
    18.     value: "master"
    19.   kubeletExtraArgs:
    20.     cgroupDriver: "cgroupfs"
    21. apiEndpoint:
    22.   advertiseAddress: "10.100.0.1"
    23.   bindPort: 6443
    24. ---
    25. apiVersion: kubeadm.k8s.io/v1alpha3
    26. kind: ClusterConfiguration
    27. etcd:
    28.   # one of local or external
    29.   local:
    30.     image: "k8s.gcr.io/etcd-amd64:3.2.18"
    32.     extraArgs:
    33.       listen-client-urls: "http://10.100.0.1:2379"
    34.     serverCertSANs:
    35.     -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    36.     peerCertSANs:
    37.     - "10.100.0.1"
    38.   external:
    39.     endpoints:
    40.     - "10.100.0.1:2379"
    41.     - "10.100.0.2:2379"
    42.     caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
    43.     certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
    44.     certKey: "/etcd/kubernetes/pki/etcd/etcd.key"
    45. networking:
    46.   serviceSubnet: "10.96.0.0/12"
    47.   podSubnet: "10.100.0.1/24"
    48.   dnsDomain: "cluster.local"
    49. kubernetesVersion: "v1.12.0"
    50. controlPlaneEndpoint: "10.100.0.1:6443"
    51. apiServerExtraArgs:
    52.   authorization-mode: "Node,RBAC"
    53. controlManagerExtraArgs:
    54.   node-cidr-mask-size: 20
    55. schedulerExtraArgs:
    56.   address: "10.100.0.1"
    57. apiServerExtraVolumes:
    58. - name: "some-volume"
    59.   hostPath: "/etc/some-path"
    60.   mountPath: "/etc/some-pod-path"
    61.   writable: true
    62.   pathType: File
    63. controllerManagerExtraVolumes:
    64. - name: "some-volume"
    65.   hostPath: "/etc/some-path"
    66.   mountPath: "/etc/some-pod-path"
    67.   writable: true
    68.   pathType: File
    69. schedulerExtraVolumes:
    70. - name: "some-volume"
    71.   hostPath: "/etc/some-path"
    72.   mountPath: "/etc/some-pod-path"
    73.   writable: true
    74.   pathType: File
    75. apiServerCertSANs:
    76. - "10.100.1.1"
    77. - "ec2-10-100-0-1.compute-1.amazonaws.com"
    78. certificatesDir: "/etc/kubernetes/pki"
    79. imageRepository: "k8s.gcr.io"
    80. unifiedControlPlaneImage: "k8s.gcr.io/controlplane:v1.12.0"
    81. auditPolicy:
    82.   # https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy
    83.   path: "/var/log/audit/audit.json"
    84.   logDir: "/var/log/audit"
    85.   logMaxAge: 7 # in days
    86. featureGates:
    87.   selfhosting: false
    88. clusterName: "example-cluster"

    注意：JoinConfiguration 重命名自 v1alpha2 API 中的 NodeConfiguration，而 InitConfiguration 重命名自 v1alpha2 API 中的 MasterConfiguration。

    1. kubeadm init --config ./kubeadm.yaml

    配置 Network plugin

    flannel

    注意：需要 kubeadm init 时设置 --pod-network-cidr=10.244.0.0/16

    1. kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
    1. sysctl net.bridge.bridge-nf-call-iptables=1

    calico

    注意：需要 kubeadm init 时设置 --pod-network-cidr=192.168.0.0/16

    1. kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
    2. kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

    跟 Master 一样，添加 Node 的时候也可以自定义 Kubernetes 服务的选项，格式为

    1. apiVersion: kubeadm.k8s.io/v1alpha2
    2. caCertPath: /etc/kubernetes/pki/ca.crt
    3. clusterName: kubernetes
    4. discoveryFile: ""
    5. discoveryTimeout: 5m0s
    6. discoveryToken: abcdef.0123456789abcdef
    7. discoveryTokenAPIServers:
    8. - kube-apiserver:6443
    9. discoveryTokenUnsafeSkipCAVerification: true
    10. kind: NodeConfiguration
    11. nodeRegistration:
    12.   criSocket: /var/run/dockershim.sock
    13.   name: thegopher
    14. tlsBootstrapToken: abcdef.0123456789abcdef
    15. token: abcdef.0123456789abcdef

    在把 Node 加入集群的时候，指定 NodeConfiguration 配置文件的路径

    Cloud Provider

    1. kind: MasterConfiguration
    2. apiVersion: kubeadm.k8s.io/v1alpha2
    3. apiServerExtraArgs:
    4.   cloud-provider: "{cloud}"
    5.   cloud-config: "{cloud-config-path}"
    6. apiServerExtraVolumes:
    7. - name: cloud
    8.   hostPath: "{cloud-config-path}"
    9.   mountPath: "{cloud-config-path}"
    10. controllerManagerExtraArgs:
    11.   cloud-provider: "{cloud}"
    12.   cloud-config: "{cloud-config-path}"
    13. controllerManagerExtraVolumes:
    14. - name: cloud
    15.   hostPath: "{cloud-config-path}"
    16.   mountPath: "{cloud-config-path}"

    删除安装

    1. # drain and delete the node first
    2. kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
    3. kubectl delete node <node name>
    5. # then reset kubeadm
    6. kubeadm reset

    kubeadm v1.8 开始支持动态升级，升级步骤为

    • 首先上传 kubeadm 配置，如 kubeadm config upload from-flags [flags]（使用命令行参数）或 kubeadm config upload from-file --config [config]（使用配置文件）
    • 在 master 上检查新版本 kubeadm upgrade plan， 当有新版本（如 v1.8.0）时，执行 kubeadm upgrade apply v1.8.0 升级控制平面
    • 手动 升级 CNI 插件（如果有新版本的话）
    • 添加自动证书回滚的 RBAC 策略 kubectl create clusterrolebinding kubeadm:node-autoapprove-certificate-rotation --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
    • 最后升级 kubelet
    1. $ kubectl drain $HOST --ignore-daemonsets
    3. # 升级软件包
    4. $ apt-get update
    5. $ apt-get upgrade
    6. # CentOS 上面执行 yum 升级
    7. # $ yum update
    9. $ kubectl uncordon $HOST

    kubeadm v1.7 以及以前的版本不支持动态升级，但可以手动升级。

    升级 Master

    假设你已经有一个使用 kubeadm 部署的 Kubernetes v1.6 集群，那么升级到 v1.7 的方法为：

    1. 升级安装包 apt-get upgrade && apt-get update
    2. 重启 kubelet systemctl restart kubelet
    3. 删除 kube-proxy DaemonSet KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete daemonset kube-proxy -n kube-system
    4. kubeadm init —skip-preflight-checks —kubernetes-version v1.7.1
    5. 更新 CNI 插件

    升级 Node

    1. 升级安装包 apt-get upgrade && apt-get update
    2. 重启 kubelet systemctl restart kubelet

    安全选项

    默认情况下，kubeadm 会开启 Node 客户端证书的自动批准，如果不需要的话可以选择关闭，关闭方法为

    1. $ kubectl delete clusterrole kubeadm:node-autoapprove-bootstrap
    1. $ kubectl get csr
    2. NAME                                                   AGE       REQUESTOR                 CONDITION
    3. node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   18s       system:bootstrap:878f07   Pending
    5. $ kubectl certificate approve node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ
    6. certificatesigningrequest "node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ" approved
    8. $ kubectl get csr
    9. NAME                                                   AGE       REQUESTOR                 CONDITION

    参考文档