In the general case, upgrading from etcd 2.2 to 2.3 can be a zero-downtime, rolling upgrade:- one by one, stop the etcd v2.2 processes and replace them with etcd v2.3 processes- after running all v2.3 processes, new features in v2.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Requirements

To upgrade an existing etcd deployment to 2.3, the running cluster must be 2.2 or greater. If it’s before 2.2, please upgrade to 2.2 before upgrading to 2.3.

Also, to ensure a smooth rolling upgrade, the running cluster must be healthy. You can check the health of the cluster by using the command.

Preparation

Before upgrading etcd, always test the services relying on etcd in a staging environment before deploying the upgrade to the production environment.

Mixed Versions

While upgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is only considered upgraded once all of its members are upgraded to version 2.3. Internally, etcd members negotiate with each other to determine the overall cluster version, which controls the reported version and the supported features.

Limitations

It might take up to 2 minutes for the newly upgraded member to catch up with the existing cluster when the total data size is larger than 50MB. Check the size of a recent snapshot to estimate the total data size. In other words, it is safest to wait for 2 minutes between upgrading each member.

For a much larger total data size, 100MB or more , this one-time process might take even more time. Administrators of very large etcd clusters of this magnitude can feel free to contact the etcd team before upgrading, and we’ll be happy to provide advice on the procedure.

Downgrade

If all members have been upgraded to v2.3, the cluster will be upgraded to v2.3, and downgrade from this completed state is not possible. If any single member is still v2.2, however, the cluster and its operations remains “v2.2”, and it is possible from this mixed cluster state to return to using a v2.2 etcd binary on all members.

Please backup the data directory of all etcd members to make downgrading the cluster possible even after it has been completely upgraded.

Upgrade Procedure

1. Check upgrade requirements.

Is the the cluster healthy and running v.2.2.x?

2. Stop the existing etcd process

When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:

  1. 2016-03-11 09:50:49.860319 E | rafthttp: failed to read 8211f1d0f64f3269 on stream Message (unexpected EOF)
  2. 2016-03-11 09:50:49.860335 I | rafthttp: the connection with 8211f1d0f64f3269 became inactive
  3. 2016-03-11 09:50:51.023821 W | etcdserver: cannot get the version of member 8211f1d0f64f3269 (Get http://127.0.0.1:12380/version: dial tcp 127.0.0.1:12380: getsockopt: connection refused)

It’s a good idea at this point to backup the etcd data directory to provide a downgrade path should any problems occur:

3. Drop-in etcd v2.3 binary and start the new etcd process

The new v2.3 etcd will publish its information to the cluster:

  1. 09:58:25.938673 I | etcdserver: published {Name:infra1 ClientURLs:[http://localhost:12379]} to cluster 524400597fb1d5f6

Verify that each member, and then the entire cluster, becomes healthy with the new v2.3 etcd binary:

  2. 2016-03-11 09:58:26.851854 W | etcdserver: member c02c70ede158499f has a higher version 2.3.0

4. Repeat step 2 to step 3 for all other members

5. Finish

When all members are upgraded, the cluster will report upgrading to 2.3 successfully:

  1. $ curl http://127.0.0.1:4001/version