Key Concepts

    Before diving into Fluent Bit it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor.

    • Event or Record
    • Tag
    • Timestamp
    • Match
    • Structured Message

    Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record.

    As an example consider the following content of a Syslog file:

    It contains four lines and all of them represents four independent Events.

    Internally, an Event always has two components (in an array form):

    Filtering

    In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.

    • Append specific information to the Event like an IP address or metadata.
    • Drop Events that matches certain pattern.

    Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.

    Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.

    {% hint style=”info” %} The only input plugin that does NOT assign tags is input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client. {% endhint %}

    A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the Routing section.

    Timestamp

    The Timestamp represents the time when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:

    It is the number of seconds that have elapsed since the Unix epoch.

    {% hint style=”info” %} A timestamp always exists, either set by the Input plugin or discovered through a data parsing process. {% endhint %}

    Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.

    To learn more about Tags and Matches check the Routing section.

    Structured Messages

    Source events can have or not have a structure. A structure defines a set of keys and values inside the Event message. As an example consider the following two messages:

    At a low level both are just an array of bytes, but the Structured message defines keys and values, having a structure helps to implement faster operations on data modifications.

    {% hint style=”info” %} Fluent Bit always handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called MessagePack.