Set up a GlusterFS Server

    This tutorial demonstrates how to configure GlusterFS on three server machines and install Heketi to manage your GlusterFS cluster.

    Once you have GlusterFS and Heketi set up, you can install GlusterFS on your client machine and use KubeKey to create a KubeSphere cluster with GlusterFS as a storage class.

    There are three server machines of Ubuntu 16.04 in this example with each having one attached disk.

    Note

    • Heketi will be installed on server1, which provides a RESTful management interface to manage the lifecycle of GlusterFS volumes. You can install it on a separate machine as well.

    • Attach more block storage disks to your server machine if you need more storage space.

    • Data will be saved to /dev/vdd (block device), which must be original without partitioning or formatting.

    Set up Passwordless SSH Login

    1. Log in to server1 and switch to the root user.

    2. Change the root user password:

      1. passwd

      Note

      Make sure password authentication is enabled in the file /etc/ssh/sshd_config (the value of PasswordAuthentication should be yes).

    3. Change the root user password of server2 and server3 as well.

    1. Configure your DNS or edit the /etc/hosts file on all server machines to add their hostnames and IP addresses:

      1. vi /etc/hosts
      1. # hostname loopback address
      2. 192.168.0.2  server1
      3. 192.168.0.3  server2
      4. 192.168.0.4  server3

    2. Make sure you add the above entries to the hosts file of all server machines.

    1. On server1, create a key by running the following command. Press Enter directly for all the prompts.

      1. ssh-keygen

    2. Copy the key to all GlusterFS nodes.

      1. ssh-copy-id [email protected]
      1. ssh-copy-id [email protected]
      1. ssh-copy-id [email protected]

    3. Verify that you can access all server machines from server1 through passwordless login.

      1. ssh [email protected]
      1. ssh [email protected]
      1. ssh [email protected]

    1. On server1, run the following command to install software-properties-common.

      1. apt-get install software-properties-common
      1. add-apt-repository ppa:gluster/glusterfs-7

    3. Make sure you are using the latest package.

      1. apt-get update

    4. Install the GlusterFS server.

      1. apt-get install glusterfs-server -y

    5. Make sure you run the above commands on server2 and server3 as well and verify the version on all machines.

      1. glusterfs -V

    Note

    The above commands may be slightly different if you do no install GlusterFS on Ubuntu. For more information, see the Gluster documentation.

    Load Kernel Modules

    1. Run the following commands to load three necessary kernel modules on server1.

      1. echo dm_snapshot | sudo tee -a /etc/modules
      1. echo dm_mirror | sudo tee -a /etc/modules

    2. Intall thin-provisioning-tools.

      1. apt-get -y install thin-provisioning-tools

    3. Make sure you run the above commands on server2 and server3 as well.

    1. Run the following command on server1 to add other nodes and create a cluster.

      1. gluster peer probe server2
      1. gluster peer probe server3

    2. Verify that all nodes in the cluster are connected successfully.

      1. gluster peer status

    3. Expected output:

      1. Number of Peers: 2
      3. Hostname: server2
      4. Uuid: e1192d6a-b65e-4ce8-804c-72d9425211a6
      5. State: Peer in Cluster (Connected)
      7. Hostname: server3
      8. Uuid: 9bd733e4-96d4-49d5-8958-6c947a2b4fa6
      9. State: Peer in Cluster (Connected)

    Install Heketi

    As GlusterFS itself does not provide a way for API calls, you can install to manage the lifecycle of GlusterFS volumes with a RESTful API for Kubernetes calls. In this way, your Kubernetes cluster can dynamically provision GlusterFS volumes. Heketi v7.0.0 will be installed in this example. For more information about available Heketi versions, see its Release Page.

    1. Download Heketi on server1.

      1. wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-v7.0.0.linux.amd64.tar.gz

      Note

      You can also install Heketi on a separate machine.

    2. Unzip the file.

      1. tar -xf heketi-v7.0.0.linux.amd64.tar.gz
      1. cd heketi
      1. cp heketi /usr/bin
      1. cp heketi-cli /usr/bin

    3. Create a Heketi service file.

      1. vi /lib/systemd/system/heketi.service
      1. [Unit]
      2. Description=Heketi Server
      3. [Service]
      4. Type=simple
      5. WorkingDirectory=/var/lib/heketi
      6. ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
      7. StandardOutput=syslog
      8. StandardError=syslog
      9. [Install]
      10. WantedBy=multi-user.target

    4. Create Heketi folders.

      1. vi /etc/heketi/heketi.json

      An example file:

      ``` { “_port_comment”: “Heketi Server Port Number”, “port”: “8080”,

      “_use_auth”: “Enable JWT authorization. Please enable for deployment”, “use_auth”: false,

      “_jwt”: “Private keys for access”, “jwt”: {

      1. "_admin": "Admin has access to all APIs",
      2. "admin": {
      3.   "key": "123456"
      4. },
      5. "_user": "User only has access to /volumes endpoint",
      6. "user": {
      7.   "key": "123456"
      8. }

      },

      “_glusterfs_comment”: “GlusterFS Configuration”, “glusterfs”: {

      1. "_executor_comment": [
      2.   "Execute plugin. Possible choices: mock, ssh",
      3.   "mock: This setting is used for testing and development.",
      4.   "      It will not send commands to any node.",
      5.   "ssh:  This setting will notify Heketi to ssh to the nodes.",
      6.   "      It will need the values in sshexec to be configured.",
      7.   "kubernetes: Communicate with GlusterFS containers over",
      8.   "            Kubernetes exec api."
      9. ],
      10. "executor": "ssh",
      12. "_sshexec_comment": "SSH username and private key file information",
      13. "sshexec": {
      14.   "keyfile": "/root/.ssh/id_rsa",
      15.   "user": "root"
      16. },
      18. "_kubeexec_comment": "Kubernetes configuration",
      19. "kubeexec": {
      20.   "host" :"https://kubernetes.host:8443",
      21.   "cert" : "/path/to/crt.file",
      22.   "insecure": false,
      23.   "user": "kubernetes username",
      24.   "password": "password for kubernetes user",
      25.   "namespace": "Kubernetes namespace",
      26.   "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
      27. },
      29. "_db_comment": "Database file name",
      30. "db": "/var/lib/heketi/heketi.db",
      31. "brick_max_size_gb" : 1024,

      “brick_min_size_gb” : 1, “max_bricks_per_volume” : 33,

    1.     "_loglevel_comment": [
    2.       "Set log level. Choices are:",
    3.       "  none, critical, error, warning, info, debug",
    4.       "Default is warning"
    5.     ],
    6.     "loglevel" : "debug"
    7.   }
    8. }
    9. ```
    11. Note
    13. The account `admin` and its `key` value must be provided when you install GlusterFS as a storage class of your KubeSphere cluster.

    1. Start Heketi.

      1. systemctl start heketi

    2. Check the status of Heketi.

      1. systemctl status heketi

      If you can see active (running), it means the installation is successful. Expected output:

      1. ● heketi.service - Heketi Server
      2.    Loaded: loaded (/lib/systemd/system/heketi.service; disabled; vendor preset: enabled)
      3.    Active: active (running) since Tue 2021-03-09 13:04:30 CST; 4s ago
      4.  Main PID: 9282 (heketi)
      5.     Tasks: 8
      6.    Memory: 6.5M
      7.       CPU: 62ms
      8.    CGroup: /system.slice/heketi.service
      9.            └─9282 /usr/bin/heketi --config=/etc/heketi/heketi.json
      11. Mar 09 13:04:30 server1 systemd[1]: Started Heketi Server.
      12. Mar 09 13:04:30 server1 heketi[9282]: Heketi v7.0.0
      13. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Loaded ssh executor
      14. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max bricks per volume set to 33
      15. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max brick size 1024 GB
      16. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 GlusterFS Application Loaded
      18. Mar 09 13:04:30 server1 heketi[9282]: Listening on port 8080

    3. Enable Heketi.

      1. systemctl enable heketi

      Expected output:

      1. Created symlink from /etc/systemd/system/multi-user.target.wants/heketi.service to /lib/systemd/system/heketi.service.

    4. Create a topology configuration file for Heketi. It contains the information of clusters, nodes, and disks added to Heketi.

      1. vi /etc/heketi/topology.json

      An example file:

      1.    {
      2.     "clusters": [
      3.        {
      4.          "nodes": [
      5.            {
      6.              "node": {
      7.                "hostnames": {
      8.                  "manage": [
      9.                    "192.168.0.2" 
      10.                 ],
      11.                 "storage": [
      12.                   "192.168.0.2" 
      13.                 ]
      14.               },
      15.               "zone": 1
      16.             },
      17.             "devices": [
      18.               "/dev/vdd" 
      19.             ]
      20.           },
      21.           {
      22.             "node": {
      23.               "hostnames": {
      24.                 "manage": [
      25.                   "192.168.0.3" 
      26.                 ],
      27.                 "storage": [
      28.                   "192.168.0.3"
      29.                 ]
      30.               },
      31.               "zone": 1
      32.             },
      33.             "devices": [
      34.               "/dev/vdd" 
      35.             ]
      36.           },
      37.           {
      38.              "node": {
      39.                "hostnames": {
      40.                  "manage": [
      41.                    "192.168.0.4"
      42.                 ],
      43.                 "storage": [
      44.                   "192.168.0.4"  
      45.                 ]
      46.               },
      47.               "zone": 1
      48.             },
      49.             "devices": [
      50.               "/dev/vdd"
      51.             ]
      52.           }
      53.         ]
      54.       }
      55.     ]
      56.   }

      Note

      • Replace the IP addresses above with your own.
      • Add your own disk name for devices.

    5. Load the Heketi JSON file.

      1. export HEKETI_CLI_SERVER=http://localhost:8080
      1. heketi-cli topology load --json=/etc/heketi/topology.json

      Expected output:

      1. Creating cluster ... ID: 2d9e11adede04fe6d07cb81c5a1a7ea4
      2.    Allowing file volumes on cluster.
      3.    Allowing block volumes on cluster.
      4.    Creating node 192.168.0.2 ... ID: 0a9f240ab6fd96ea014948c5605be675
      5.        Adding device /dev/vdd ... OK
      6.    Creating node 192.168.0.3 ... ID: 2468086cadfee8ef9f48bc15db81c88a
      7.        Adding device /dev/vdd ... OK
      8.    Creating node 192.168.0.4 ... ID: 4c21b33d5c32029f5b7dc6406977ec34
      9.        Adding device /dev/vdd ... OK

    6. The above output displays both your cluster ID and node ID. Run the following command to view your cluster information.