It is advised that you run the restore from your local host or a jump box/bastion where your cluster yaml, rke statefile, and kubeconfig are stored. You will need and kubectl CLI utilities installed locally.
Prepare by creating 3 new nodes to be the target for the restored Rancher instance. We recommend that you start with fresh nodes and a clean state. For clarification on the requirements, review the .
Alternatively you can re-use the existing nodes after clearing Kubernetes and Rancher configurations. This will destroy the data on these nodes. See Node Cleanup for the procedure.
You must restore each of your etcd nodes to the same snapshot. Copy the snapshot you’re using from one of your nodes to the others before running the etcd snapshot-restore
command.
IMPORTANT: Before starting the restore make sure all the Kubernetes services on the old cluster nodes are stopped. We recommend powering off the nodes to be sure.
As of RKE v0.2.0, snapshots could be saved in an S3 compatible backend. To restore your cluster from the snapshot stored in S3 compatible backend, you can skip this step and retrieve the snapshot in . Otherwise, you will need to place the snapshot directly on one of the etcd nodes.
Use your original rancher-cluster.yml
and rancher-cluster.rkestate
files. If they are not stored in a version control system, it is a good idea to back them up before making any changes.
If the replaced or cleaned nodes have been configured with new IP addresses, modify the rancher-cluster.yml
file to ensure the address and optional internal_address fields reflect the new addresses.
IMPORTANT: You should not rename the
rancher-cluster.yml
orrancher-cluster.rkestate
files. It is important that the filenames match each other.
You will now use the RKE command-line tool with the rancher-cluster.yml
and the rancher-cluster.rkestate
configuration files to restore the etcd database and bring up the cluster on the new nodes.
Restoring from a Local Snapshot
When restoring etcd from a local snapshot, the snapshot is assumed to be located on the target node in the directory .
Restoring from a Snapshot in S3
Available as of RKE v0.2.0
When restoring etcd from a snapshot located in an S3 compatible backend, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
Options for rke etcd snapshot-restore
S3 specific options are only available for RKE v0.2.0+.
Testing the Cluster
Once RKE completes it will have created a credentials file in the local directory. Configure kubectl
to use the credentials file and check on the state of the cluster. See Installing and Configuring kubectl for details.
Check Kubernetes Pods
Wait for the pods running in kube-system
, ingress-nginx
and the rancher
pod in cattle-system
to return to the Running
state.
Finishing Up
Rancher should now be running and available to manage your Kubernetes clusters. > IMPORTANT: Remember to save your updated RKE config (rancher-cluster.yml
) state file (rancher-cluster.rkestate
) and kubectl
credentials (kube_config_rancher-cluster.yml
) files in a safe place for future maintenance for example in a version control system.