Multi-node Installation

    This section gives you an overview of a single-master multi-node installation, including the concept, KubeKey and steps. For information about HA installation, refer to , Installing on Public Cloud and .

    Concept

    A multi-node cluster is composed of at least one master node and one worker node. You can use any node as the taskbox to carry out the installation task. You can add additional nodes based on your needs (e.g. for high availability) both before and after the installation.

    • Master. A master node generally hosts the control plane that controls and manages the whole system.
    • Worker. Worker nodes run the actual applications deployed on them.

    Please see the requirements for hardware and operating system shown below. To get started with multi-node installation in this tutorial, you need to prepare at least three hosts according to the following requirements. It is possible to install the on two nodes if they have sufficient resources.

    Note

    The path is mainly used to store the container data, and will gradually increase in size during use and operation. In the case of a production environment, it is recommended that /var/lib/docker should mount a drive separately.

    Node requirements

    • All nodes must be accessible through SSH.
    • Time synchronization for all nodes.
    • sudo/curl/openssl should be used in all nodes.
    • docker can be installed by yourself or by KubeKey.

    Note

    docker must be installed in advance if you want to deploy KubeSphere in an offline environment.

    KubeKey can install Kubernetes and KubeSphere together. The dependency that needs to be installed may be different based on the Kubernetes version to be installed. You can refer to the list below to see if you need to install relevant dependencies on your node in advance.

    Network and DNS requirements

    • If your network configuration uses firewall rules or security groups, you must ensure infrastructure components can communicate with each other through specific ports. It’s recommended that you turn off the firewall or follow the guide Port Requirements.

    Tip

    • It’s recommended that your OS be clean (without any other software installed). Otherwise, there may be conflicts.
    • A registry mirror (booster) is recommended to be prepared if you have trouble downloading images from dockerhub.io. See and Configure registry mirrors for the Docker daemon.

    This example includes three hosts as below with the master node serving as the taskbox.

    Step 2: Download KubeKey

    Follow the step below to download KubeKey.

    Download KubeKey from its or use the following command directly.

    Run the following command first to make sure you download KubeKey from the correct zone.

    1. export KKZONE=cn

    Run the following command to download KubeKey:

    1. curl -sfL https://get-kk.kubesphere.io | VERSION=v1.1.0 sh -

    Note

    After you download KubeKey, if you transfer it to a new machine also with poor network connections to Googleapis, you must run export KKZONE=cn again before you proceed with the steps below.

    Note

    The commands above download the latest release (v1.1.0) of KubeKey. You can change the version number in the command to download a specific version.

    Make kk executable:

    1. chmod +x kk

    Command:

    Note

    • Recommended Kubernetes versions for KubeSphere v3.1.0: v1.17.9, v1.18.8, v1.19.8 and v1.20.4. If you do not specify a Kubernetes version, KubeKey will install Kubernetes v1.19.8 by default. For more information about supported Kubernetes versions, see Support Matrix.

    • If you do not add the flag --with-kubesphere in the command in this step, KubeSphere will not be deployed unless you install it using the addons field in the configuration file or add this flag again when you use ./kk create cluster later.

    • If you add the flag --with-kubesphere without specifying a KubeSphere version, the latest version of KubeSphere will be installed.

    Here are some examples for your reference:

    • You can create an example configuration file with default configurations. You can also specify the file with a different filename, or in a different folder.

      1. ./kk create config [-f ~/myfolder/abc.yaml]

    • You can specify a KubeSphere version that you want to install (e.g. --with-kubesphere v3.1.0).

      1. ./kk create config --with-kubesphere [version]

    2. Edit the configuration file

    A default file will be created if you do not change the name. Edit the file and here is an example of the configuration file of a multi-node cluster with one master node.

    Note

    To customize Kubernetes related parameters, refer to Kubernetes Cluster Configurations.

    1. spec:
    2.   hosts:
    3.   - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, user: ubuntu, password: Testing123}
    4.   - {name: node1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: ubuntu, password: Testing123}
    5.   - {name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: ubuntu, password: Testing123}
    6.   roleGroups:
    7.     etcd:
    8.     - master
    9.     master:
    10.     - master
    11.     worker:
    12.     - node1
    13.   controlPlaneEndpoint:
    14.     domain: lb.kubesphere.local
    15.     address: ""
    16.     port: "6443"

    Hosts

    List all your machines under hosts and add their detailed information as above.

    name: The hostname of the instance.

    address: The IP address you use for the connection between the taskbox and other instances through SSH. This can be either the public IP address or the private IP address depending on your environment. For example, some cloud platforms provide every instance with a public IP address which you use to access instances through SSH. In this case, you can input the public IP address for this field.

    internalAddress: The private IP address of the instance.

    At the same time, you must provide the login information used to connect to each instance. Here are some examples:

    • For password login:

      Note

      In this tutorial, port 22 is the default port of SSH so you do not need to add it in the YAML file. Otherwise, you need to add the port number after the IP address as above.

      1. hosts:
      2.   - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, password: Testing123}

    • For passwordless login with SSH keys:

      1. hosts:

    Tip

    • Before you install KubeSphere, you can use the information provided under hosts (for example, IP addresses and passwords) to test the network connection between the taskbox and other instances using SSH.
    • Make sure port 6443 is not being used by other services before the installation. Otherwise, it may cause conflicts as the default port of the API server is 6443.

    roleGroups

    • etcd: etcd node names
    • master: Master node names
    • worker: Worker node names

    controlPlaneEndpoint (for HA installation only)

    The controlPlaneEndpoint is where you provide your external load balancer information for an HA cluster. You need to prepare and configure the external load balancer if and only if you need to install multiple master nodes. Please note that the address and port should be indented by two spaces in config-sample.yaml, and address should be your load balancer’s IP address. See HA Configurations for details.

    addons

    You can customize persistent storage plugins (e.g. NFS Client, Ceph RBD, and GlusterFS) by specifying storage under the field addons in config-sample.yaml. For more information, see Persistent Storage Configurations.

    KubeKey will install to provision LocalPV for development and testing environment by default, which is convenient for new users. In this example of multi-node installation, the default storage class (local volume) is used. For production, you can use NFS/Ceph/GlusterFS/CSI or commercial products as persistent storage solutions.

    Tip

    • You can enable the multi-cluster feature by editing the configuration file. For more information, see .
    • You can also select the components you want to install. For more information, see Enable Pluggable Components. For an example of a complete config-sample.yaml file, see .

    When you finish editing, save the file.

    1. ./kk create cluster -f config-sample.yaml

    Note

    You need to change config-sample.yaml above to your own file if you use a different name.

    The whole installation process may take 10-20 minutes, depending on your machine and network.

    4. Verify the installation

    When the installation finishes, you can see the content as follows:

    Now, you will be able to access the web console of KubeSphere at <NodeIP>:30880 with the default account and password (admin/[[email protected]](https://kubesphere.io/cdn-cgi/l/email-protection)).

    Note

    To access the console, you may need to configure port forwarding rules depending on your environment. Please also make sure port 30880 is opened in your security group.

    Enable kubectl Autocompletion

    KubeKey doesn’t enable kubectl autocompletion. See the content below and turn it on:

    Note

    Make sure bash-autocompletion is installed and works.

    1. # Install bash-completion
    2. apt-get install bash-completion
    4. # Source the completion script in your ~/.bashrc file
    5. echo 'source <(kubectl completion bash)' >>~/.bashrc
    7. kubectl completion bash >/etc/bash_completion.d/kubectl

    Detailed information can be found here.