Pipelines

    A pipeline is used to transform a single log line, its labels, and itstimestamp. A pipeline is comprised of a set of stages. There are 4 types ofstages:

    1. Parsing stages parse the current log line and extract data out of it. Theextracted data is then available for use by other stages.
    2. Action stages take extracted data from previous stages and do somethingwith them. Actions can:
      1. Add or modify existing labels to the log line
      2. Change the timestamp of the log line
      3. Change the content of the log line
      4. Create a metric based on the extracted data
    3. Filtering stages optionally apply a subset of stages based on somecondition.

    Typical pipelines will start with a parsing stage (such as aregex or stage) to extract datafrom the log line. Then, a series of action stages will be present to dosomething with that extracted data. The most common action stage will be alabels stage to turn extracted data into a label.

    A common stage will also be the stage to selectivelyapply stages based on the current labels.

    Note that pipelines can not currently be used to deduplicate logs; Loki willreceive the same log line multiple times if, for example:

    1. Two scrape configs read from the same file
    2. Duplicate log lines in a file are sent through a pipeline. Deduplication isnot done.

    However, Loki will perform some deduplication at query time for logs that havethe exact same nanosecond timestamp, labels, and log contents.

    The following sections further describe the types that are accessible to eachstage (although not all may be used):

    Label Set

    The current set of labels for the log line. Initialized to be the set of labelsthat were scraped along with the log line. The label set is only modified by anaction stage, but filtering stages read from it.

    The final label set will be index by Loki and can be used for queries.

    Extracted Map

    A collection of key-value pairs extracted during a parsing stage. Subsequentstages operate on the extracted map, either transforming them or taking actionwith them. At the end of a pipeline, the extracted map is discarded; for aparsing stage to be useful, it must always be paired with at least one actionstage.

    The extracted map is initialized with the same set of initial labels that werescraped along with the log line. This initial data allows for taking action onthe values of labels inside pipeline stages that only manipulate the extractedmap. For example, log entries tailed from files have the label whosevalue is the file path that was tailed. When a pipeline executes for that logentry, the initial extracted map would contain filename using the same valueas the label.

    Log Timestamp

    The final value for the timestamp is sent to Loki.

    Log Line

    The current log line, represented as text. Initialized to be the text thatPromtail scraped. Action stages can modify this value.

    The final value for the log line is sent to Loki as the text content for thegiven log entry.

    Stages

    Parsing stages:

    • : Extract data by parsing the log line using the standard CRI format.
    • regex: Extract data using a regular expression.
    • : Extract data by parsing the log line as JSON.

    Transform stages:

    • template: Use Go templates to modify extracted data.
    • : Set the timestamp value for the log entry.
    • output: Set the log line text.
    • : Update the label set for the log entry.
    • metrics: Calculate metrics based on extracted data.

    Filtering stages: