Upgrade etcd from 3.1 to 3.2

    In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

    • one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
    • after running all v3.2 processes, new features in v3.2 are available to the cluster

    Before , read through the rest of this guide to prepare.

    NOTE: When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

    Highlighted breaking changes in 3.2.

    Changed default snapshot-count value

    Higher --snapshot-count holds more Raft entries in memory until snapshot, thus causing . Since leader retains latest Raft entries for longer, a slow follower has more time to catch up before leader snapshot. --snapshot-count is a tradeoff between higher memory usage and better availabilities of slow followers.

    Since v3.2, the default value of --snapshot-count has changed from from 10,000 to 100,000.

    Changed gRPC dependency (>=3.2.10)

    3.2.10 or later now requires v1.7.5 (<=3.2.9 requires v1.2.1).

    Deprecated grpclog.Logger

    grpclog.Logger has been deprecated in favor of grpclog.LoggerV2. clientv3.Logger is now grpclog.LoggerV2.

    Before

    After

    1. import "github.com/coreos/etcd/clientv3"
    2. import "google.golang.org/grpc/grpclog"
    3. clientv3.SetLogger(grpclog.NewLoggerV2(os.Stderr, os.Stderr, os.Stderr))
    5. // log.New above cannot be used (not implement grpclog.LoggerV2 interface)
    Deprecated grpc.ErrClientConnTimeout

    Previously, grpc.ErrClientConnTimeout error is returned on client dial time-outs. 3.2 instead returns context.DeadlineExceeded (see ).

    Before

    1. // expect dial time-out on ipv4 blackhole
    2. _, err := clientv3.New(clientv3.Config{
    3.     Endpoints:   []string{"http://254.0.0.1:12345"},
    4.     DialTimeout: 2 * time.Second
    5. })
    6. if err == grpc.ErrClientConnTimeout {
    7.     // handle errors

    After

    1. _, err := clientv3.New(clientv3.Config{
    2.     Endpoints:   []string{"http://254.0.0.1:12345"},
    3.     DialTimeout: 2 * time.Second
    4. })
    5. if err == context.DeadlineExceeded {
    6.     // handle errors

    Changed maximum request size limits (>=3.2.10)

    3.2.10 and 3.2.11 allow custom request size limits in server side. >=3.2.12 allows custom request size limits for both server and client side. In previous versions(v3.2.10, v3.2.11), client response size was limited to only 4 MiB.

    Server-side request limits can be configured with --max-request-bytes flag:

    1. # limits request size to 1.5 KiB
    2. etcd --max-request-bytes 1536
    4. # client writes exceeding 1.5 KiB will be rejected
    5. etcdctl put foo [LARGE VALUE...]
    6. # etcdserver: request is too large
    1. import "github.com/coreos/etcd/embed"
    2. import "github.com/coreos/etcd/etcdserver/api/v3rpc/rpctypes"
    4. // limit requests to 5 MiB
    5. cfg := embed.NewConfig()
    6. cfg.MaxRequestBytes = 5 * 1024 * 1024
    8. // client writes exceeding 5 MiB will be rejected
    9. _, err := cli.Put(ctx, "foo", [LARGE VALUE...])
    10. err == rpctypes.ErrRequestTooLarge

    If not specified, server-side limit defaults to 1.5 MiB.

    Client-side request limits must be configured based on server-side limits.

    1. # limits request size to 1 MiB
    2. etcd --max-request-bytes 1048576

    If not specified, client-side send limit defaults to 2 MiB (1.5 MiB + gRPC overhead bytes) and receive limit to math.MaxInt32. Please see clientv3 godoc for more detail.

    Changed raw gRPC client wrappers

    3.2.12 or later changes the function signatures of clientv3 gRPC client wrapper. This change was needed to support .

    Before and after

    1. -func NewKVFromKVClient(remote pb.KVClient) KV {
    2. +func NewKVFromKVClient(remote pb.KVClient, c *Client) KV {
    4. -func NewClusterFromClusterClient(remote pb.ClusterClient) Cluster {
    5. +func NewClusterFromClusterClient(remote pb.ClusterClient, c *Client) Cluster {
    7. -func NewLeaseFromLeaseClient(remote pb.LeaseClient, keepAliveTimeout time.Duration) Lease {
    8. +func NewLeaseFromLeaseClient(remote pb.LeaseClient, c *Client, keepAliveTimeout time.Duration) Lease {
    10. -func NewMaintenanceFromMaintenanceClient(remote pb.MaintenanceClient) Maintenance {
    11. +func NewMaintenanceFromMaintenanceClient(remote pb.MaintenanceClient, c *Client) Maintenance {
    12. -func NewWatchFromWatchClient(wc pb.WatchClient) Watcher {
    13. +func NewWatchFromWatchClient(wc pb.WatchClient, c *Client) Watcher {

    Changed clientv3.Lease.TimeToLive API

    Previously, clientv3.Lease.TimeToLive API returned lease.ErrLeaseNotFound on non-existent lease ID. 3.2 instead returns TTL=-1 in its response and no error (see #7305).

    Before

    1. // when leaseID does not exist
    2. resp, err := TimeToLive(ctx, leaseID)
    4. err == lease.ErrLeaseNotFound

    After

    1. // when leaseID does not exist
    2. resp, err := TimeToLive(ctx, leaseID)
    3. resp.TTL == -1
    4. err == nil

    Moved clientv3.NewFromConfigFile to clientv3.yaml.NewConfig

    clientv3.NewFromConfigFile is moved to yaml.NewConfig.

    Before

    1. import "github.com/coreos/etcd/clientv3"
    2. clientv3.NewFromConfigFile

    After

    1. import clientv3yaml "github.com/coreos/etcd/clientv3/yaml"
    2. clientv3yaml.NewConfig

    Change in --listen-peer-urls and --listen-client-urls

    3.2 now rejects domains names for --listen-peer-urls and --listen-client-urls (3.1 only prints out warnings), since domain name is invalid for network interface binding. Make sure that those URLs are properly formated as scheme://IP:port.

    See for more contexts.

    Upgrade requirements

    To upgrade an existing etcd deployment to 3.2, the running cluster must be 3.1 or greater. If it’s before 3.1, please upgrade to 3.1 before upgrading to 3.2.

    Also, to ensure a smooth rolling upgrade, the running cluster must be healthy. Check the health of the cluster by using the etcdctl endpoint health command before proceeding.

    Preparation

    Before beginning, . Should something go wrong with the upgrade, it is possible to use this backup to downgrade back to existing etcd version. Please note that the snapshot command only backs up the v3 data. For v2 data, see .

    Mixed versions

    While upgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is only considered upgraded once all of its members are upgraded to version 3.2. Internally, etcd members negotiate with each other to determine the overall cluster version, which controls the reported version and the supported features.

    Limitations

    Note: If the cluster only has v3 data and no v2 data, it is not subject to this limitation.

    If the cluster is serving a v2 data set larger than 50MB, each newly upgraded member may take up to two minutes to catch up with the existing cluster. Check the size of a recent snapshot to estimate the total data size. In other words, it is safest to wait for 2 minutes between upgrading each member.

    For a much larger total data size, 100MB or more , this one-time process might take even more time. Administrators of very large etcd clusters of this magnitude can feel free to contact the etcd team before upgrading, and we’ll be happy to provide advice on the procedure.

    Downgrade

    If all members have been upgraded to v3.2, the cluster will be upgraded to v3.2, and downgrade from this completed state is not possible. If any single member is still v3.1, however, the cluster and its operations remains “v3.1”, and it is possible from this mixed cluster state to return to using a v3.1 etcd binary on all members.

    Please of all etcd members to make downgrading the cluster possible even after it has been completely upgraded.

    This example shows how to upgrade a 3-member v3.1 ectd cluster running on a local machine.

    1. Check upgrade requirements

    Is the cluster healthy and running v3.1.x?

    1. $ ETCDCTL_API=3 etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
    2. localhost:2379 is healthy: successfully committed proposal: took = 6.600684ms
    3. localhost:22379 is healthy: successfully committed proposal: took = 8.540064ms
    4. localhost:32379 is healthy: successfully committed proposal: took = 8.763432ms
    6. $ curl http://localhost:2379/version
    7. {"etcdserver":"3.1.7","etcdcluster":"3.1.0"}

    2. Stop the existing etcd process

    When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:

    It’s a good idea at this point to backup the etcd data to provide a downgrade path should any problems occur:

    1. $ etcdctl snapshot save backup.db

    3. Drop-in etcd v3.2 binary and start the new etcd process

    The new v3.2 etcd will publish its information to the cluster:

    1. 2017-04-27 14:14:25.363225 I | etcdserver: published {Name:s1 ClientURLs:[http://localhost:2379]} to cluster a9ededbffcb1b1f1

    Verify that each member, and then the entire cluster, becomes healthy with the new v3.2 etcd binary:

    1. $ ETCDCTL_API=3 /etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
    2. localhost:22379 is healthy: successfully committed proposal: took = 5.540129ms
    3. localhost:32379 is healthy: successfully committed proposal: took = 7.321771ms
    4. localhost:2379 is healthy: successfully committed proposal: took = 10.629901ms

    Upgraded members will log warnings like the following until the entire cluster is upgraded. This is expected and will cease after all etcd cluster members are upgraded to v3.2:

    1. 2017-04-27 14:15:17.071804 W | etcdserver: member c89feb932daef420 has a higher version 3.2.0
    2. 2017-04-27 14:15:21.073110 W | etcdserver: the local etcd version 3.1.7 is not up-to-date
    3. 2017-04-27 14:15:21.073142 W | etcdserver: member 6d4f535bae3ab960 has a higher version 3.2.0
    4. 2017-04-27 14:15:21.073157 W | etcdserver: the local etcd version 3.1.7 is not up-to-date
    5. 2017-04-27 14:15:21.073164 W | etcdserver: member c89feb932daef420 has a higher version 3.2.0

    4. Repeat step 2 to step 3 for all other members

    5. Finish

    When all members are upgraded, the cluster will report upgrading to 3.2 successfully:

    1. 2017-04-27 14:15:54.536901 N | etcdserver/membership: updated the cluster version from 3.1 to 3.2
    2. 2017-04-27 14:15:54.537035 I | etcdserver/api: enabled capabilities for version 3.2
    1. $ ETCDCTL_API=3 /etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
    2. localhost:2379 is healthy: successfully committed proposal: took = 2.312897ms
    3. localhost:32379 is healthy: successfully committed proposal: took = 2.517902ms