Orchestrator is a tool for managing MySQL replication topologies, including automated failover. It can detect master failure and initiate a recovery in a matter of seconds.
For the most part, Vitess is agnostic to the actions of Orchestrator, which operates below Vitess at the MySQL level. That means you can pretty much in the normal way, with just a few additions as described below.
Orchestrator needs to know some things from the Vitess side, like the tablet aliases and whether semisync is enforced with async fallback disabled. We pass this information by telling Orchestrator to execute certain queries that return local metadata from a non-replicated table, as seen in our sample orchestrator.conf.json:
Vitess also needs to know the identity of the master for each shard. This is necessary in case of a failover.
It is important to ensure that orchestrator has access to so that orchestrator can trigger the change in topology via the command.
"echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log",
],
VTTablet configuration
Luckily, Vitess already knows everything about all the MySQL instances that comprise your cluster. So we provide a mechanism for tablets to self-register with the Orchestrator API, configured by the following vttablet parameters:
orc_discover_interval
: How often (e.g. 60s) to ping Orchestrator’s HTTP API endpoint to tell it we exist. 0 means never.
Not only does this relieve you from the initial seeding of addresses into Orchestrator, it also means new instances will be discovered immediately, and the topology will automatically repopulate even if Orchestrator’s backing store is wiped out. Note that Orchestrator will forget stale instances after a configurable timeout.