我的书签
添加书签
移除书签Elasticsearch Example
Prepare to modify configuration file
- conf/SortClusterConfig.conf, sink configuration of all sort tasks.
- conf/sid_es_v3.conf, data source configuration example of a sort task, the file name is same with sort task name in SortClusterConfig.conf.
Example: conf/SortClusterConfig.conf
Example: conf/sid_es_v3.conf
Modify configuration file: SortClusterConfig.conf
- Get cluster configuration data from the file:SortClusterConfig.conf in classpath, it can not support online updating.
- Get cluster configuration data from InlongManager URL, it can support online updating.
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| clusterName | Y | String | NA | inlong-sort-standalone cluster id |
| sortTasks | Y | JsonArray<SortTaskConfig> | NA | Sort task list |
Modify configuration: idParams of Elasticsearch sort task
| Parameter | Required | DefaultValue | Remark |
|---|---|---|---|
| inlongGroupId | Y | NA | inlongGroupId |
| inlongStreamId | Y | NA | inlongStreamId |
| separator | Y | NA | separator of Inlong datastream in data source |
| fieldNames | Y | NA | field name list of Elasticsearch index, separated by space. |
| indexNamePattern | Y | NA | index name pattern of Elasticsearch,date time variable include {yyyyMMdd},{yyyyMMddHH},{yyyyMMddHHmm}. |
| contentOffset | Y | NA | field index offset of source content |
| fieldOffset | Y | NA | offset of Elasticsearch index field name list |
Modify configuration: sinkParams of Elasticsearch sort task
- The file name include sort task name plus the postfix “.conf”.
- Get the configuration data from the file in classpath, it can not support online updating.
- Get the configuration data from InlongManager URL, it can support online updating.
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| sortClusterName | Y | String | NA | inlong-sort-standalone cluster id |
| sortTaskId | Y | String | NA | Sort task name |
| cacheZones | Y | JsonObject<String, JsonObject> | NA | Cache cluster list, Map<cacheClusterName, CacheCluster> |
Modify configuration: CacheCluster
Modify configuration: Topic
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| topic | Y | String | NA | cache topic name |
| partitionCnt | Y | Integer | NA | cache topic partition count |
| topicProperties | N | Map<String,String> | NA | Cache topic configuration |
