list_to_map

The following table describes the configuration options used to generate target keys for the mappings.

Usage

The following example shows how to test the usage of the list_to_map processor before using the processor on your own source.

Create a source file named logs_json.log. Because the file source reads each line in the .log file as an event, the object list appears as one line even though it contains multiple objects:

copy

Next, create a pipeline.yaml file that uses the logs_json.log file as the source by pointing to the .log file’s correct path:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        value_key: "value"
        flatten: true
  sink:
    - stdout:

copy

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": "val-a",
  "b": "val-b1",
}

The following example pipeline.yaml file shows the list_to_map processor when set to a specified target, mymap:

copy

The generated map appears under the target key:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "mymap": {
    "a": "val-a",
    "b": "val-b1",
    "c": "val-c"
  }
}

The follow example pipeline.yaml file shows the list_to_map processor with no value_key specified. Because key is set to name, the processor extracts the object names to use as keys in the map.

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        flatten: true
  sink:
    - stdout:

copy

The values from the generated map appear as original objects from the .log source, as shown in the following example response:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        value_key: "value"
        flatten: true
        flattened_element: "last"
  sink:
    - stdout:

copy

The processor maps object b to value val-b2 because val-b2 is the last element in object b, as shown in the following output:

{
  "mylist": [
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": "val-a",
  "b": "val-b2",
  "c": "val-c"
}

The following example pipeline.yaml file sets flatten to false, causing the processor to output values from the generated map as a list:

copy

Some objects in the response may have more than one element in their values, as shown in the following response:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": [
    "val-a"
  ],
  "b": [
    "val-b1",
    "val-b2"
  ],
  "c": [
    "val-c"