Tabletserver YAML parameters

This issue is an expansion of #5791, and covers the details of how we’ll use yaml to implement a hierarchy of parameters for initializing and customizing TabletServer components within a process.

Using the yaml approach, we intend to solve the following problems:

  • Multiple tablet servers need to exist within a process.
  • Badly named command line parameters: The new yaml specifications will have simpler and easier to remember parameter names.
  • Sections for improved readability.
  • Unreasonable default values: The new specs will introduce a simplified approach that will save the user from tweaking too many variables without knowing their consequences.
  • Repetition: If multiple TabletServers have to share the same parameter values, they should not be repeated.
  • Backward compatibility: The system must be backward compatible.

We will introduce two new command-line options that will be specified like this:

We’ll provide predefined “defaults” files that people can use as starting points. We’ll start with something simple like small.yaml, medium.yaml, and large.yaml.

We explored the use of protobufs for converting to and from yaml. However, the protobuf JSON implementation, which was required to correctly parse enums, was unable to preserve the original values for sub-objects. This was a showstopper for making defaults work.

The existing data structure will be converted into a struct with the new names and appropriate JSON tags.

The following section lists all the properties that can be specified in a yaml file along with the existing defaults.

During implementation, we’ll need to decide between creating a universal protobuf that represents this structure vs converting the original one into something more usable.

Note that certain properties (like tabletID) are only applicable to tablet-specific files.

  1. tabletID: zone-1234
  3. init:
  4.   dbName:            # init_db_name_override
  5.   keyspace:          # init_keyspace
  6.   shard:             # init_shard
  7.   tabletType:        # init_tablet_type
  8.   timeoutSeconds: 60 # init_timeout
  10. db:
  11.   socket:     # db_socket
  12.   host:       # db_host
  13.   port: 0     # db_port
  14.   charSet:    # db_charset
  15.   flags: 0    # db_flags
  16.   flavor:     # db_flavor
  17.   sslCa:      # db_ssl_ca
  18.   sslCaPath:  # db_ssl_ca_path
  19.   sslCert:    # db_ssl_cert
  20.   sslKey:     # db_ssl_key
  21.   serverName: # db_server_name
  22.   connectTimeoutMilliseconds: 0 # db_connect_timeout_ms
  23.   app:
  24.     user: vt_app      # db_app_user
  25.     password:         # db_app_password
  26.     useSsl: true      # db_app_use_ssl
  27.     preferSocket: true
  28.   dba:
  29.     user: vt_dba      # db_dba_user
  30.     password:         # db_dba_password
  31.     useSsl: true      # db_dba_use_ssl
  32.     preferSocket: true
  33.   filtered:
  34.     password:         # db_filtered_password
  35.     useSsl: true      # db_filtered_use_ssl
  36.     preferSocket: true
  38.     user: vt_repl     # db_repl_user
  39.     password:         # db_repl_password
  40.     useSsl: true      # db_repl_use_ssl
  41.     preferSocket: true
  42.   appdebug:
  43.     user: vt_appdebug # db_appdebug_user
  44.     password:         # db_appdebug_password
  45.     useSsl: true      # db_appdebug_use_ssl
  46.     preferSocket: true
  47.   allprivs:
  48.     user: vt_allprivs # db_allprivs_user
  49.     password:         # db_allprivs_password
  50.     useSsl: true      # db_allprivs_use_ssl
  51.     preferSocket: true
  53. oltpReadPool:
  54.   size: 16                 # queryserver-config-pool-size
  55.   timeoutSeconds: 0        # queryserver-config-query-pool-timeout
  56.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  57.   prefillParallelism: 0    # queryserver-config-pool-prefill-parallelism
  58.   maxWaiters: 50000        # queryserver-config-query-pool-waiter-cap
  60. olapReadPool:
  61.   size: 200                # queryserver-config-stream-pool-size
  62.   timeoutSeconds: 0        # queryserver-config-query-pool-timeout
  63.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  64.   prefillParallelism: 0    # queryserver-config-stream-pool-prefill-parallelism
  65.   maxWaiters: 0
  67. txPool:
  68.   size: 20                 # queryserver-config-transaction-cap
  69.   timeoutSeconds: 1        # queryserver-config-txpool-timeout
  70.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  71.   prefillParallelism: 0    # queryserver-config-transaction-prefill-parallelism
  72.   maxWaiters: 50000        # queryserver-config-txpool-waiter-cap
  74. oltp:
  76.   txTimeoutSeconds: 30    # queryserver-config-transaction-timeout
  77.   warnRows: 0             # queryserver-config-warn-result-size
  79. hotRowProtection:
  80.   mode: disable|dryRun|enable # enable_hot_row_protection, enable_hot_row_protection_dry_run
  81.   # Default value is same as txPool.size.
  82.   maxQueueSize: 20            # hot_row_protection_max_queue_size
  83.   maxGlobalQueueSize: 1000    # hot_row_protection_max_global_queue_size
  84.   maxConcurrency: 5           # hot_row_protection_concurrent_transactions
  86. consolidator: enable|disable|notOnMaster # enable-consolidator, enable-consolidator-replicas
  87. heartbeatIntervalMilliseconds: 0         # heartbeat_enable, heartbeat_interval
  88. shutdownGracePeriodSeconds: 0            # transaction_shutdown_grace_period
  89. passthroughDML: false                    # queryserver-config-passthrough-dmls
  90. streamBufferSize: 32768                  # queryserver-config-stream-buffer-size
  91. queryCacheSize: 5000                     # queryserver-config-query-cache-size
  92. schemaReloadIntervalSeconds: 1800        # queryserver-config-schema-reload-time
  93. watchReplication: false                  # watch_replication_stream
  94. terseErrors: false                       # queryserver-config-terse-errors
  95. messagePostponeParallelism: 4            # queryserver-config-message-postpone-cap
  96. cacheResultFields: true                  # enable-query-plan-field-caching
  99. # The following flags are currently not supported.
  100. # enforce_strict_trans_tables
  101. # queryserver-config-strict-table-acl
  102. # queryserver-config-enable-table-acl-dry-run
  103. # queryserver-config-acl-exempt-acl
  104. # enable-tx-throttler
  105. # tx-throttler-config
  106. # tx-throttler-healthcheck-cells
  107. # enable_transaction_limit
  108. # enable_transaction_limit_dry_run
  109. # transaction_limit_per_user
  110. # transaction_limit_by_username
  111. # transaction_limit_by_principal
  113. # transaction_limit_by_subcomponent

There are also other global parameters. VTTablet has a total of 405 flags. We may have to later move some of them into these yamls.

We’ll use the unified tabletenv.TabletConfig data structure to load defaults as well as tablet-specific values. The following changes will be made:

  • TabletConfig will be changed to match the above specifications.
  • The existing flags will be changed to update the values in a global TabletConfig.
  • In case of type mismatch (like time.Duration vs a “Seconds” variable), an extra step will be performed after parsing to convert the flag variables into TabletConfig members.
  • The code will be changed to honor the new TabletConfig members.
  • After the flag.Parse step, a copy of the global Config will be used as input to load the defaults file. This means that any command line flags that are not overridden in the yaml files will be preserved. This behavior is needed to support backward compatibility in case we decide to move more flags into the yaml.
  • For each tablet entry, we create a copy of the result TabletConfig and read the tablet yaml into those, which will set the tablet-specific values. This will then be used to instantiate a TabletServer.

The exact format of the tablets.yaml file is not fully finalized. Our options are to allow a list of files where each is for a single tablet, or, to require only one file containing a dictionary of tablet-specific overrides.

<< VTTablet with RBR mode