Requirements

Make sure ceph-common and xfsprogs packages are installed on SLE worker nodes.

Using the Ceph Driver with RKE

The resources below are fully compatible with RKE based clusters, but there is a need to do an additional kubelet configuration for RKE.

On RKE clusters, the kubelet component is running in a Docker container and doesn’t have access to the host’s kernel modules as rbd and libceph by default.

To solve this limitation, you can either run on worker nodes, or configure the kubelet containers to automatically mount the /lib/modules directory from the host into the container.

For the kubelet configuration, put the following lines into the cluster.yml file prior to RKE cluster provisioning. You can also modify the cluster.yml later in the Rancher UI by clicking on Edit Cluster > Edit as YAML and restarting the worker nodes.

For more information about the extra_binds directive, refer to this section.

Installing the ceph-csi driver on an RKE2 cluster

For more information about the ceph-csi-rbd chart, refer to this page.

  1. ceph mon dump

Read its output:

  1. dumped monmap epoch 3
  2. epoch 3
  3. fsid 79179d9d-98d8-4976-ab2e-58635caa7235
  4. last_changed 2021-02-11T10:56:42.110184+0000
  5. created 2021-02-11T10:56:22.913321+0000
  6. min_mon_release 15 (octopus)
  7. 0: [v2:10.85.8.118:3300/0,v1:10.85.8.118:6789/0] mon.a
  8. 1: [v2:10.85.8.123:3300/0,v1:10.85.8.123:6789/0] mon.b
  9. 2: [v2:10.85.8.124:3300/0,v1:10.85.8.124:6789/0] mon.c

Later you’ll need the fsid and mon addresses values.

Install the ceph-csi Driver Using Helm

Run these commands:

  1. helm repo add ceph-csi https://ceph.github.io/csi-charts
  2. helm repo update
  3. helm search repo ceph-csi -l
  4. helm inspect values ceph-csi/ceph-csi-rbd > ceph-csi-rbd-values.yaml

Modify the ceph-csi-rbd-values.yaml file and keep there only the required changes:

  1. # ceph-csi-rbd-values.yaml
  2. csiConfig:
  3.   - clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"
  4.     monitors:
  5.       - "10.85.8.118:6789"
  6.       - "10.85.8.123:6789"
  7.       - "10.85.8.124:6789"
  8. provisioner:
  9.   name: provisioner
  10.   replicaCount: 2

Make sure the ceph monitors are reachable from the RKE2 cluster, for example, by ping.

  1. kubectl create namespace ceph-csi-rbd
  2. helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml
  3. kubectl rollout status deployment ceph-csi-rbd-provisioner -n ceph-csi-rbd
  4. helm status ceph-csi-rbd -n ceph-csi-rbd

in case you’d like to modify the configuration directly via Helm, you may adapt the ceph-csi-rbd-values.yaml file and call:

Creating RBD Ceph Resources

  1. # Create a ceph pool:
  2. ceph osd pool create myPool 64 64
  4. # Create a block device pool:
  5. rbd pool init myPool
  7. # Create a block device image:
  8. rbd create -s 2G myPool/image
  10. # Create a block device user and record the key:
  11. ceph auth get-or-create-key client.myPoolUser mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=myPool" | tr -d '\n' | base64
  12. QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
  14. # Encode the ceph user myPoolUser into a bash64 hash:
  15. echo "myPoolUser" | tr -d '\n' | base64
  16. bXlQb29sVXNlcg==
  18. # Create a block device admin user and record the key:
  19. QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
  21. # Encode the ceph user myPoolAdmin into a bash64 hash:
  22. echo "myPoolAdmin" | tr -d '\n' | base64
  23. bXlQb29sQWRtaW4=

Configure RBD Ceph Access Secrets

For static RBD provisioning (the image within the ceph pool must exist), run these commands:

  1. cat > ceph-user-secret.yaml << EOF
  2. apiVersion: v1
  3. kind: Secret
  4. metadata:
  5.   name: ceph-user
  6.   namespace: default
  7. type: kubernetes.io/rbd
  8. data:
  9.   userID: bXlQb29sVXNlcg==
  10.   userKey: QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
  11. EOF

Admin Account

For dynamic RBD provisioning (used for automatic image creation within a given ceph pool), run these commands:

  1. cat > ceph-admin-secret.yaml << EOF
  2. apiVersion: v1
  3. kind: Secret
  4. metadata:
  5.   name: ceph-admin
  6.   namespace: default
  7. type: kubernetes.io/rbd
  8. data:
  9.   userID: bXlQb29sQWRtaW4=
  10.   userKey: QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
  11. EOF
  13. kubectl apply -f ceph-admin-secret.yaml

Create RBD Testing Resources

  1. # pod
  2. cat > ceph-rbd-pod-inline.yaml << EOF
  3. apiVersion: v1
  4. kind: Pod
  5. metadata:
  6.   name: ceph-rbd-pod-inline
  7. spec:
  8.   containers:
  9.   - name: ceph-rbd-pod-inline
  10.     image: busybox
  11.     command: ["sleep", "infinity"]
  12.     volumeMounts:
  13.     - mountPath: /mnt/ceph_rbd
  14.       name: volume
  15.   volumes:
  16.   - name: volume
  17.     rbd:
  18.       monitors:
  19.       - 10.85.8.118:6789
  20.       - 10.85.8.123:6789
  21.       - 10.85.8.124:6789
  22.       pool: myPool
  23.       image: image
  24.       user: myPoolUser
  25.       secretRef:
  26.         name: ceph-user
  27.       fsType: ext4
  28.       readOnly: false
  29. EOF
  31. kubectl apply -f ceph-rbd-pod-inline.yaml
  32. kubectl get pod
  33. kubectl exec pod/ceph-rbd-pod-inline -- df -k | grep rbd

Using RBD in Persistent Volumes

  1. # pod-pvc-pv
  2. cat > ceph-rbd-pod-pvc-pv-allinone.yaml << EOF
  3. apiVersion: v1
  4. kind: PersistentVolume
  5. metadata:
  6.   name: ceph-rbd-pv
  7. spec:
  8.   capacity:
  9.     storage: 2Gi
  10.   accessModes:
  11.     - ReadWriteOnce
  12.   rbd:
  13.     monitors:
  14.     - 10.85.8.123:6789
  15.     - 10.85.8.124:6789
  16.     pool: myPool 
  17.     image: image
  18.     user: myPoolUser 
  19.     secretRef:
  20.       name: ceph-user
  21.     fsType: ext4
  22.     readOnly: false
  23. ---
  24. kind: PersistentVolumeClaim
  26. metadata:
  27.   name: ceph-rbd-pvc
  28. spec:
  29.   accessModes:
  30.   - ReadWriteOnce
  31.   resources:
  32.     requests:
  33.       storage: 2Gi
  34. ---
  35. apiVersion: v1
  36. kind: Pod
  37. metadata:
  38.   name: ceph-rbd-pod-pvc-pv
  39. spec:
  40.   containers:
  41.   - name: ceph-rbd-pod-pvc-pv
  42.     image: busybox
  43.     command: ["sleep", "infinity"]
  44.     volumeMounts:
  45.     - mountPath: /mnt/ceph_rbd
  46.       name: volume
  47.   volumes:
  48.   - name: volume
  49.     persistentVolumeClaim:
  50.       claimName: ceph-rbd-pvc
  51. EOF
  53. kubectl apply -f ceph-rbd-pod-pvc-pv-allinone.yaml
  54. kubectl get pv,pvc,pod
  55. kubectl exec pod/ceph-rbd-pod-pvc-pv -- df -k | grep rbd

RKE2 Server/Master Provisioning

  1. sudo su
  2. curl -sfL https://get.rke2.io | sh -
  3. systemctl enable --now rke2-server
  5. cat > /root/.bashrc << EOF
  6. export PATH=$PATH:/var/lib/rancher/rke2/bin/
  7. export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
  8. EOF
  10. cat /var/lib/rancher/rke2/server/node-token
  11. token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
  1. mkdir -p /etc/rancher/rke2/
  3. cat > /etc/rancher/rke2/config.yaml << EOF
  4. server: https://10.100.103.23:9345
  5. token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
  6. EOF
  8. curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -
  9. systemctl enable --now rke2-agent.service

The cluster can be imported into Rancher from the Rancher UI by clicking Global/Add Cluster > Other Cluster. Then run the provided kubectl command on the server/master node.

Tested Versions

OS for running RKE2 nodes: JeOS SLE15-SP2 with installed kernel-default-5.3.18-24.49

  1. kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-22T12:00:00Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7+rke2r1", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-20T01:50:52Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"}
  5. helm version
  6. version.BuildInfo{Version:"3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.12"}

Kubernetes version on RKE2 cluster: v1.19.7+rke2r1

Troubleshooting

In case you are using SUSE’s ceph-rook based on SES7, it might be useful to expose the monitors on hostNetwork by editing rook-1.4.5/ceph/cluster.yaml and setting spec.network.hostNetwork=true.

Also for operating the ceph-rook cluster, it is useful to deploy a toolbox on the Kubernetes cluster where ceph-rook is provisioned by kubectl apply -f rook-1.4.5/ceph/toolbox.yaml Then all the ceph related commands can be executed in the toolbox pod, for example, by running kubectl exec -it -n rook-ceph rook-ceph-tools-686d8b8bfb-2nvqp -- bash

Operating with the ceph - basic commands:

  1. ceph osd pool stats
  2. ceph osd pool delete myPool myPool --yes-i-really-really-mean-it
  3. rbd list -p myPool
  4. > csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38
  5. > image

Delete the image: rbd rm csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 -p myPool

CephFS commands in rook toolbox:

  1. ceph -s
  2. ceph fs ls
  3. ceph fs fail cephfs
  4. ceph fs rm cephfs --yes-i-really-mean-it