Loki

    The Fluent Bit loki built-in output plugin allows you to send your log or events to a Loki service. It support data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.

    Loki store the record logs inside Streams, a stream is defined by a set of labels, at least one label is required.

    Fluent Bit implements a flexible mechanism to set labels by using fixed key/value pairs of text but also allowing to set as labels certain keys that exists as part of the records that are being processed. Consider the following JSON record (pretty printed for readability):

    If you decide that your Loki Stream will be composed by two labels called job and the value of the record key called stream , your labels configuration properties might look as follows:

    1. [OUTPUT]
    2.     name   loki
    3.     match  *
    4.     labels job=fluentbit, $sub['stream']

    As you can see the label job has the value fluentbit and the second label is configured to access the nested map called sub targeting the value of the key stream . Note that the second label name must starts with a , that means that’s a Record Accessor pattern so it provide you the ability to retrieve values from nested maps by using the key names.

    1. job="fluentbit", stream="stdout"

    Another feature of Labels management is the ability to provide custom key names, using the same record accessor pattern we can specify the key name manually and let the value to be populated automatically at runtime, e.g:

    1. [OUTPUT]
    2.     name   loki
    3.     match  *

    When processing that new configuration, the internal labels will be:

    The additional configuration property called label_keys allow to specify multiple record keys that needs to be placed as part of the outgoing Stream Labels, yes, this is a similar feature than the one explained above in the labels property. Consider this as another way to set a record key in the Stream, but with the limitation that you cannot use a custom name for the key value.

    The following configuration examples generate the same Stream Labels:

    1. [OUTPUT]
    2.     name       loki
    3.     match      *
    4.     labels     job=fluentbit
    5.     label_keys $sub['stream']

    the above configuration accomplish the same than this one:

    1. [OUTPUT]
    2.     name   loki
    3.     match  *
    4.     labels job=fluentbit, $sub['stream']
    1. job="fluentbit", stream="stdout"

    Kubernetes & Labels

    Note that if you are running in a Kubernetes environment, you might want to enable the option which will auto-populate the streams with the Pod labels for you. Consider the following configuration:

    Based in the JSON example provided above, the internal stream labels will be:

    1. job="fluentbit", team="Santiago Wanderers"

    This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:

    • Networking Setup: timeouts, keepalive and source address
    • : all about TLS configuration and certificates

    Note that all options mentioned in the articles above must be enabled in the plugin configuration in question.

    The following configuration example, will emit a dummy example record and ingest it on Loki . Copy and paste the following content in a file called out_loki.conf:

    1. [SERVICE]
    2.     log_level info
    4. [INPUT]
    5.     name      dummy
    6.     dummy     {"key": 1, "sub": {"stream": "stdout", "id": "some id"}, "kubernetes": {"labels": {"team": "Santiago Wanderers"}}}
    7.     samples   1
    9. [OUTPUT]
    10.     name                   loki
    11.     match                  *
    12.     host                   127.0.0.1
    13.     port                   3100
    14.     labels                 job=fluentbit    
    15.     label_keys             $sub['stream']
    1. $ fluent-bit -c out_loki.conf

    Fluent Bit output: