Upgrade etcd from 3.3 to 3.4

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before , read through the rest of this guide to prepare.

NOTE: When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Highlighted breaking changes in 3.4.

Make `ETCDCTL_API=3 etcdctl` default

ETCDCTL_API=3 is now the default.

Make `etcd --enable-v2=false` default

is now the default.

This means, unless etcd --enable-v2=true is specified, etcd v3.4 server would not serve v2 API requests.

If v2 API were used, make sure to enable v2 API in v3.4:

-etcd
+etcd --enable-v2=true

Other HTTP APIs will still work (e.g. [CLIENT-URL]/metrics, [CLIENT-URL]/health, v3 gRPC gateway).

Deprecated `etcd --ca-file` and `etcd --peer-ca-file` flags

--ca-file and --peer-ca-file flags are deprecated; they have been deprecated since v2.1.

-etcd --ca-file ca-client.crt
+etcd --trusted-ca-file ca-client.crt

-etcd --peer-ca-file ca-peer.crt
+etcd --peer-trusted-ca-file ca-peer.crt

Deprecated `grpc.ErrClientConnClosing` error

grpc.ErrClientConnClosing has been deprecated in gRPC >= 1.10.

import (
+    "go.etcd.io/etcd/clientv3"
    "google.golang.org/grpc"
+    "google.golang.org/grpc/codes"
+    "google.golang.org/grpc/status"
)
_, err := kvc.Get(ctx, "a")
-if err == grpc.ErrClientConnClosing {
+if clientv3.IsConnCanceled(err) {
// or
+s, ok := status.FromError(err)
+if ok {
+  if s.Code() == codes.Canceled

Require `grpc.WithBlock` for client dial

uses an asynchronous resolver to pass endpoints to the gRPC dial function. As a result, v3.4 client requires grpc.WithBlock dial option to wait until the underlying connection is up.

import (
    "time"
    "go.etcd.io/etcd/clientv3"
+    "google.golang.org/grpc"
)
+// "grpc.WithBlock()" to block until the underlying connection is up
ccfg := clientv3.Config{
  Endpoints:            []string{"localhost:2379"},
  DialTimeout:          time.Second,
+ DialOptions:          []grpc.DialOption{grpc.WithBlock()},
  DialKeepAliveTime:    time.Second,
  DialKeepAliveTimeout: 500 * time.Millisecond,
}

Deprecating `etcd_debugging_mvcc_db_total_size_in_bytes` Prometheus metrics

v3.4 promotes etcd_debugging_mvcc_db_total_size_in_bytes Prometheus metrics to etcd_mvcc_db_total_size_in_bytes, in order to encourage etcd storage monitoring.

etcd_debugging_mvcc_db_total_size_in_bytes is still served in v3.4 for backward compatibilities. It will be completely deprecated in v3.5.

-etcd_debugging_mvcc_db_total_size_in_bytes
+etcd_mvcc_db_total_size_in_bytes