BookKeeper 是一个冗余的日志存储系统,Pulsar 用它来持久化存储所有消息。

Broker

NameDescription默认值
enablePersistentTopicsWhether persistent topics are enabled on the brokertrue
enableNonPersistentTopicsWhether non-persistent topics are enabled on the brokertrue
functionsWorkerEnabledWhether the Pulsar Functions worker service is enabled in the brokerfalse
zookeeperServersZookeeper quorum connection string
configurationStoreServers配置存储连接字符串(以逗号分隔的列表)
brokerServicePortBroker data port6650
brokerServicePortTlsBroker data port for TLS6651
webServicePortPort to use to server HTTP request8080
webServicePortTlsPort to use to server HTTPS request8443
webSocketServiceEnabledEnable the WebSocket API service in brokerfalse
bindAddressHostname or IP address the service binds on, default is 0.0.0.0.0.0.0.0
advertisedAddressHostname or IP address the service advertises to the outside world. If not set, the value of InetAddress.getLocalHost().getHostName() is used.
clusterNameName of the cluster to which this broker belongs to
brokerDeduplicationEnabledSets the default behavior for message deduplication in the broker. If enabled, the broker will reject messages that were already stored in the topic. This setting can be overridden on a per-namespace basis.false
brokerDeduplicationMaxNumberOfProducersThe maximum number of producers for which information will be stored for deduplication purposes.10000
brokerDeduplicationEntriesIntervalThe number of entries after which a deduplication informational snapshot is taken. A larger interval will lead to fewer snapshots being taken, though this would also lengthen the topic recovery time (the time required for entries published after the snapshot to be replayed).1000
brokerDeduplicationProducerInactivityTimeoutMinutesThe time of inactivity (in minutes) after which the broker will discard deduplication information related to a disconnected producer.360
dispatchThrottlingRatePerReplicatorInMsgThe default messages per second dispatch throttling-limit for every replicator in replication. The value of 0 means disabling replication message dispatch-throttling0
dispatchThrottlingRatePerReplicatorInByteThe default bytes per second dispatch throttling-limit for every replicator in replication. The value of 0 means disabling replication message-byte dispatch-throttling0
zooKeeperSessionTimeoutMillisZookeeper session timeout in milliseconds30000
brokerShutdownTimeoutMsTime to wait for broker graceful shutdown. After this time elapses, the process will be killed60000
skipBrokerShutdownOnOOMFlag to skip broker shutdown when broker handles Out of memory error.false
backlogQuotaCheckEnabledEnable backlog quota check. Enforces action on topic when the quota is reachedtrue
backlogQuotaCheckIntervalInSecondsHow often to check for topics that have reached the quota60
backlogQuotaDefaultLimitGBThe default per-topic backlog quota limit-1
allowAutoTopicCreationEnable topic auto creation if a new producer or consumer connectedtrue
allowAutoTopicCreationTypeThe topic type (partitioned or non-partitioned) that is allowed to be automatically created.Partitioned
allowAutoSubscriptionCreationEnable subscription auto creation if a new consumer connectedtrue
defaultNumPartitionsThe number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned1
brokerDeleteInactiveTopicsEnabledEnable the deletion of inactive topicstrue
brokerDeleteInactiveTopicsFrequencySecondsHow often to check for inactive topics60
messageExpiryCheckIntervalInMinutesHow frequently to proactively check and purge expired messages5
brokerServiceCompactionMonitorIntervalInSecondsInterval between checks to see if topics with compaction policies need to be compacted60
activeConsumerFailoverDelayTimeMillisHow long to delay rewinding cursor and dispatching messages when active consumer is changed.1000
clientLibraryVersionCheckEnabledEnable check for minimum allowed client library versionfalse
clientLibraryVersionCheckAllowUnversionedAllow client libraries with no version informationtrue
statusFilePathPath for the file used to determine the rotation status for the broker when responding to service discovery health checks
preferLaterVersionsIf true, (and ModularLoadManagerImpl is being used), the load manager will attempt to use only brokers running the latest software version (to minimize impact to bundles)false
tlsEnabledEnable TLSfalse
tlsCertificateFilePathTLS证书文件的路径
tlsKeyFilePathTLS私钥文件的路径
tlsTrustCertsFilePathPath for the trusted TLS certificate file
tlsAllowInsecureConnectionAccept untrusted TLS certificate from clientfalse
tlsProtocolsSpecify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- TLSv1.2, TLSv1.1,
tlsCiphersSpecify the tls cipher the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
ttlDurationDefaultInSecondsThe default ttl for namespaces if ttl is not configured at namespace policies.0
tokenSecretKeyConfigure the secret key to be used to validate auth tokens. The key can be specified like: tokenSecretKey=data:base64,xxxxxxxxx or tokenSecretKey=file:///my/secret.key
tokenPublicKeyConfigure the public key to be used to validate auth tokens. The key can be specified like: tokenPublicKey=data:base64,xxxxxxxxx or tokenPublicKey=file:///my/secret.key
tokenPublicAlgConfigure the algorithm to be used to validate auth tokens. This can be any of the asymettric algorithms supported by Java JWT ()RS256
tokenAuthClaimSpecify which of the token’s claims will be used as the authentication “principal” or “role”. The default “sub” claim will be used if this is left blank
tokenAudienceClaimThe token audience “claim” name, e.g. “aud”, that will be used to get the audience from token. If not set, audience will not be verified.
tokenAudienceThe token audience stands for this broker. The field tokenAudienceClaim of a valid token, need contains this.
maxUnackedMessagesPerConsumerMax number of unacknowledged messages allowed to receive messages by a consumer on a shared subscription. Broker will stop sending messages to consumer once, this limit reaches until consumer starts acknowledging messages back. Using a value of 0, is disabling unackeMessage limit check and consumer can receive messages without any restriction50000
maxUnackedMessagesPerSubscriptionMax number of unacknowledged messages allowed per shared subscription. Broker will stop dispatching messages to all consumers of the subscription once this limit reaches until consumer starts acknowledging messages back and unack count reaches to limit/2. Using a value of 0, is disabling unackedMessage-limit check and dispatcher can dispatch messages without any restriction200000
subscriptionRedeliveryTrackerEnabledEnable subscription message redelivery trackertrue
maxConcurrentLookupRequestMax number of concurrent lookup request broker allows to throttle heavy incoming lookup traffic50000
maxConcurrentTopicLoadRequestMax number of concurrent topic loading request broker allows to control number of zk-operations5000
authenticationEnabledEnable authenticationfalse
authenticationProvidersAutentication provider name list, which is comma separated list of class names
authorizationEnabledEnforce authorizationfalse
superUserRolesRole names that are treated as “super-user”, meaning they will be able to do all admin operations and publish/consume from all topics
brokerClientAuthenticationPluginAuthentication settings of the broker itself. Used when the broker connects to other brokers, either in same or other clusters
brokerClientAuthenticationParameters
athenzDomainNamesSupported Athenz provider domain names(comma separated) for authentication
exposePreciseBacklogInPrometheusEnable expose the precise backlog stats, set false to use published counter and consumed counter to calculate, this would be more efficient but may be inaccurate.false
bookkeeperClientAuthenticationPluginAuthentication plugin to use when connecting to bookies
bookkeeperClientAuthenticationParametersNameBookKeeper auth plugin implementatation specifics parameters name and values
bookkeeperClientAuthenticationParameters
bookkeeperClientTimeoutInSecondsTimeout for BK add / read operations30
bookkeeperClientSpeculativeReadTimeoutInMillisSpeculative reads are initiated if a read request doesn’t complete within a certain time Using a value of 0, is disabling the speculative reads0
bookkeeperClientHealthCheckEnabledEnable bookies health check. Bookies that have more than the configured number of failure within the interval will be quarantined for some time. During this period, new ledgers won’t be created on these bookiestrue
bookkeeperClientHealthCheckIntervalSeconds60
bookkeeperClientHealthCheckErrorThresholdPerInterval5
bookkeeperClientHealthCheckQuarantineTimeInSeconds1800
bookkeeperClientRackawarePolicyEnabledEnable rack-aware bookie selection policy. BK will chose bookies from different racks when forming a new bookie ensembletrue
bookkeeperClientRegionawarePolicyEnabledEnable region-aware bookie selection policy. BK will chose bookies from different regions and racks when forming a new bookie ensemble. If enabled, the value of bookkeeperClientRackawarePolicyEnabled is ignoredfalse
bookkeeperClientReorderReadSequenceEnabledEnable/disable reordering read sequence on reading entries.false
bookkeeperClientIsolationGroupsEnable bookie isolation by specifying a list of bookie groups to choose from. Any bookie outside the specified groups will not be used by the broker
bookkeeperClientSecondaryIsolationGroupsEnable bookie secondary-isolation group if bookkeeperClientIsolationGroups doesn’t have enough bookie available.
bookkeeperClientMinAvailableBookiesInIsolationGroupsMinimum bookies that should be available as part of bookkeeperClientIsolationGroups else broker will include bookkeeperClientSecondaryIsolationGroups bookies in isolated list.
bookkeeperEnableStickyReadsEnable/disable having read operations for a ledger to be sticky to a single bookie. If this flag is enabled, the client will use one single bookie (by preference) to read all entries for a ledger.true
managedLedgerDefaultEnsembleSizeNumber of bookies to use when creating a ledger2
managedLedgerDefaultWriteQuorumNumber of copies to store for each message2
managedLedgerDefaultAckQuorumNumber of guaranteed copies (acks to wait before write is complete)2
managedLedgerCacheSizeMBAmount of memory to use for caching data payload in managed ledger. This memory is allocated from JVM direct memory and it’s shared across all the topics running in the same broker. By default, uses 1/5th of available direct memory
managedLedgerCacheCopyEntriesWhether we should make a copy of the entry payloads when inserting in cachefalse
managedLedgerCacheEvictionWatermarkThreshold to which bring down the cache level when eviction is triggered0.9
managedLedgerCacheEvictionFrequencyConfigure the cache eviction frequency for the managed ledger cache (evictions/sec)100.0
managedLedgerCacheEvictionTimeThresholdMillisAll entries that have stayed in cache for more than the configured time, will be evicted1000
managedLedgerCursorBackloggedThresholdConfigure the threshold (in number of entries) from where a cursor should be considered ‘backlogged’ and thus should be set as inactive.1000
managedLedgerDefaultMarkDeleteRateLimitRate limit the amount of writes per second generated by consumer acking the messages1.0
managedLedgerMaxEntriesPerLedgerMax number of entries to append to a ledger before triggering a rollover. A ledger rollover is triggered on these conditions:
  • Either the max rollover time has been reached
  • or max entries have been written to the ledged and at least min-time has passed
50000
managedLedgerMinLedgerRolloverTimeMinutesMinimum time between ledger rollover for a topic10
managedLedgerMaxLedgerRolloverTimeMinutesMaximum time before forcing a ledger rollover for a topic240
managedLedgerCursorMaxEntriesPerLedgerMax number of entries to append to a cursor ledger50000
managedLedgerCursorRolloverTimeInSecondsMax time before triggering a rollover on a cursor ledger14400
managedLedgerMaxUnackedRangesToPersistMax number of “acknowledgment holes” that are going to be persistently stored. When acknowledging out of order, a consumer will leave holes that are supposed to be quickly filled by acking all the messages. The information of which messages are acknowledged is persisted by compressing in “ranges” of messages that were acknowledged. After the max number of ranges is reached, the information will only be tracked in memory and messages will be redelivered in case of crashes.1000
autoSkipNonRecoverableDataSkip reading non-recoverable/unreadable data-ledger under managed-ledger’s list.It helps when data-ledgers gets corrupted at bookkeeper and managed-cursor is stuck at that ledger.false
loadBalancerEnabledEnable load balancertrue
loadBalancerPlacementStrategyStrategy to assign a new bundle weightedRandomSelection
loadBalancerReportUpdateThresholdPercentagePercentage of change to trigger load report update10
loadBalancerReportUpdateMaxIntervalMinutesmaximum interval to update load report15
loadBalancerHostUsageCheckIntervalMinutesFrequency of report to collect1
loadBalancerSheddingIntervalMinutesLoad shedding interval. Broker periodically checks whether some traffic should be offload from some over-loaded broker to other under-loaded brokers30
loadBalancerSheddingGracePeriodMinutesPrevent the same topics to be shed and moved to other broker more that once within this timeframe30
loadBalancerBrokerMaxTopicsUsage threshold to allocate max number of topics to broker50000
loadBalancerBrokerUnderloadedThresholdPercentageUsage threshold to determine a broker as under-loaded1
loadBalancerBrokerOverloadedThresholdPercentageUsage threshold to determine a broker as over-loaded85
loadBalancerResourceQuotaUpdateIntervalMinutesInterval to update namespace bundle resource quotat15
loadBalancerBrokerComfortLoadLevelPercentageUsage threshold to determine a broker is having just right level of load65
loadBalancerAutoBundleSplitEnabledenable/disable namespace bundle auto splitfalse
loadBalancerNamespaceBundleMaxTopicsmaximum topics in a bundle, otherwise bundle split will be triggered1000
loadBalancerNamespaceBundleMaxSessionsmaximum sessions (producers + consumers) in a bundle, otherwise bundle split will be triggered1000
loadBalancerNamespaceBundleMaxMsgRatemaximum msgRate (in + out) in a bundle, otherwise bundle split will be triggered1000
loadBalancerNamespaceBundleMaxBandwidthMbytesmaximum bandwidth (in + out) in a bundle, otherwise bundle split will be triggered100
loadBalancerNamespaceMaximumBundlesmaximum number of bundles in a namespace128
replicationMetricsEnabledEnable replication metricstrue
replicationConnectionsPerBrokerMax number of connections to open for each broker in a remote cluster More connections host-to-host lead to better throughput over high-latency links.16
replicationProducerQueueSizeReplicator producer queue size1000
replicatorPrefixReplicator prefix used for replicator producer name and cursor name pulsar.repl
replicationTlsEnabledEnable TLS when talking with other clusters to replicate messagesfalse
defaultRetentionTimeInMinutesDefault message retention time
defaultRetentionSizeInMBDefault retention size0
keepAliveIntervalSecondsHow often to check whether the connections are still alive30
loadManagerClassNameName of load manager to useorg.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl
supportedNamespaceBundleSplitAlgorithmsSupported algorithms name for namespace bundle split[range_equally_divide,topic_count_equally_divide]
defaultNamespaceBundleSplitAlgorithmDefault algorithm name for namespace bundle splitrange_equally_divide
managedLedgerOffloadDriverDriver to use to offload old data to long term storage (Possible values: S3)
managedLedgerOffloadMaxThreadsMaximum number of thread pool threads for ledger offloading2
managedLedgerUnackedRangesOpenCacheSetEnabledUse Open Range-Set to cache unacknowledged messagestrue
managedLedgerOffloadDeletionLagMsDelay between a ledger being successfully offloaded to long term storage and the ledger being deleted from bookkeeper14400000
managedLedgerOffloadAutoTriggerSizeThresholdBytesThe number of bytes before triggering automatic offload to long term storage-1 (disabled)
s3ManagedLedgerOffloadRegionFor Amazon S3 ledger offload, AWS region
s3ManagedLedgerOffloadBucketFor Amazon S3 ledger offload, Bucket to place offloaded ledger into
s3ManagedLedgerOffloadServiceEndpointFor Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing)
s3ManagedLedgerOffloadMaxBlockSizeInBytesFor Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)67108864
s3ManagedLedgerOffloadReadBufferSizeInBytesFor Amazon S3 ledger offload, Read buffer size in bytes (1MB by default)1048576
s3ManagedLedgerOffloadRoleFor Amazon S3 ledger offload, provide a role to assume before writing to s3
s3ManagedLedgerOffloadRoleSessionNameFor Amazon S3 ledger offload, provide a role session name when using a rolepulsar-s3-offload
maxMessageSizeSet the maximum size of a message.5 MB

Client

The CLI tool can be used to publish messages to Pulsar and consume messages from Pulsar topics. This tool can be used in lieu of a client library.

NameDescription默认值
webServiceUrl群集的 web URL。http://localhost:8080/
brokerServiceUrl集群的Pulsar 协议地址。
authPlugin身份认证插件。
authParams群集的身份认证参数, 逗号分隔的字符串。
useTls是否在群集中强制执行 TLS 验证。false
tlsAllowInsecureConnection
tlsTrustCertsFilePath

Log4j

Name默认值
pulsar.root.loggerWARN,CONSOLE
pulsar.log.dirlogs
pulsar.log.filepulsar.log
log4j.rootLogger${pulsar.root.logger}
log4j.appender.CONSOLEorg.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.ThresholdDEBUG
log4j.appender.CONSOLE.layoutorg.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern%d{ISO8601} - %-5p - [%t:%C{1}@%L] - %m%n
log4j.appender.ROLLINGFILEorg.apache.log4j.DailyRollingFileAppender
log4j.appender.ROLLINGFILE.ThresholdDEBUG
log4j.appender.ROLLINGFILE.File${pulsar.log.dir}/${pulsar.log.file}
log4j.appender.ROLLINGFILE.layoutorg.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
log4j.appender.TRACEFILEorg.apache.log4j.FileAppender
log4j.appender.TRACEFILE.ThresholdTRACE
log4j.appender.TRACEFILE.Filepulsar-trace.log
log4j.appender.TRACEFILE.layoutorg.apache.log4j.PatternLayout
log4j.appender.TRACEFILE.layout.ConversionPattern%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n

Log4j shell

Name默认值
bookkeeper.root.loggerERROR,CONSOLE
log4j.rootLogger${bookkeeper.root.logger}
log4j.appender.CONSOLEorg.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.ThresholdDEBUG
log4j.appender.CONSOLE.layoutorg.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern%d{ABSOLUTE} %-5p %m%n
log4j.logger.org.apache.zookeeperERROR
log4j.logger.org.apache.bookkeeperERROR
log4j.logger.org.apache.bookkeeper.bookie.BookieShellINFO

WebSocket

NameDescription默认值
configurationStoreServers
zooKeeperSessionTimeoutMillis30000
serviceUrl
serviceUrlTls
brokerServiceUrl
brokerServiceUrlTls
webServicePort8080
webServicePortTls8443
bindAddress0.0.0.0
clusterName
authenticationEnabledfalse
authenticationProviders
authorizationEnabledfalse
superUserRoles
brokerClientAuthenticationPlugin
brokerClientAuthenticationParameters
tlsEnabledfalse
tlsAllowInsecureConnectionfalse
tlsCertificateFilePath
tlsKeyFilePath
tlsTrustCertsFilePath

Pulsar proxy

NameDescription默认值
zookeeperServersZooKeeper quorum 连接字符串(以逗号分隔的列表)
configurationStoreServers配置存储连接字符串(以逗号分隔的列表)
zookeeperSessionTimeoutMsZooKeeper会话超时(以毫秒为单位)30000
servicePort用于服务器二进制Protobuf请求的端口6650
servicePortTls用于服务器二进制Protobuf TLS请求的端口6651
statusFilePath在响应服务发现健康检查时,用于确定代理实例的轮换状态的文件的路径
authenticationEnabled是否为Pulsar代理启用身份验证false
authenticateMetricsEndpointWhether the ‘/metrics’ endpoint requires authentication. Defaults to true. ‘authenticationEnabled’ must also be set for this to take effect.true
authenticationProviders身份验证提供者名称列表(以逗号分隔的类名列表)
authorizationEnabled是否由Pulsar代理强制执行授权false
authorizationProvider授权提供程序的完全限定类名org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
brokerClientAuthenticationPluginPulsar代理使用的身份验证插件,用于对Pulsar brokers进行身份验证
brokerClientAuthenticationParametersPulsar代理用于对Pulsar Brokers进行身份验证的参数
brokerClientTrustCertsFilePathPulsar代理用于对Pulsar Brokers进行身份验证的可信证书的路径
superUserRoles“超级用户”的角色名,这意味着它们将能够执行所有管理
forwardAuthorizationCredentialsWhether client authorization credentials are forwared to the broker for re-authorization. Authentication must be enabled via authenticationEnabled=true for this to take effect.false
maxConcurrentInboundConnectionsMax concurrent inbound connections. The proxy will reject requests beyond that.10000
maxConcurrentLookupRequestsMax concurrent outbound connections. The proxy will error out requests beyond that.50000
tlsEnabledInProxy是否为代理启用TLSfalse
tlsEnabledWithBroker与Pulsar Brokers通信时是否启用TLSfalse
tlsCertificateFilePathTLS证书文件的路径
tlsKeyFilePathTLS私钥文件的路径
tlsTrustCertsFilePath受信任的TLS证书pem文件的路径
tlsHostnameVerificationEnabled当代理与brokers建立TLS连接时是否验证主机名false
tlsRequireTrustedClientCertOnConnectWhether client certificates are required for TLS. Connections are rejected if the client certificate isn’t trusted.false
tlsProtocolsSpecify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- , TLSv1.1, TLSv1
tlsCiphersSpecify the tls cipher the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
tokenSecretKeyConfigure the secret key to be used to validate auth tokens. The key can be specified like: tokenSecretKey=data:base64,xxxxxxxxx or tokenSecretKey=file:///my/secret.key
tokenPublicKeyConfigure the public key to be used to validate auth tokens. The key can be specified like: tokenPublicKey=data:base64,xxxxxxxxx or tokenPublicKey=file:///my/secret.key
tokenPublicAlgConfigure the algorithm to be used to validate auth tokens. This can be any of the asymettric algorithms supported by Java JWT (https://github.com/jwtk/jjwt#signature-algorithms-keys)RS256
tokenAuthClaimSpecify the token claim that will be used as the authentication “principal” or “role”. The “subject” field will be used if this is left blank
proxyLogLevelSet the Pulsar Proxy log level.
  • If the value is set to 0, no TCP channel information is logged.
  • If the value is set to 1, only the TCP channel information and command information (without message body) are parsed and logged.
  • 0

    ZooKeeper handles a broad range of essential configuration- and coordination-related tasks for Pulsar. The default configuration file for ZooKeeper is in the conf/zookeeper.conf file in your Pulsar installation. The following parameters are available: