Buffering / Storage

    By default when Fluent Bit process data, it uses Memory as a primary and temporal place to store the record logs, but there are certain scenarios where would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.

    Starting with Fluent Bit v1.0, we introduced a new storage layer that can either work in memory or in the file system. Input plugins can be configured to use one or the other upon demand at start time.

    • Service Section
    • Input Section

    The known Service section configure a global environment for the storage layer, and then in the Input sections defines which mechanism to use.

    a Service section will look like this:

    Input Section Configuration

    Optionally, any Input plugin can configure their storage preference, the following table describe the options available:

    Key Description Default
    storage.type Specify the buffering mechanism to use. It can be memory or filesystem. memory

    The following example configure a service that offers filesystem buffering capabilities and two Input plugins being the first based in memory and the second with the filesystem:

    2.     log_Level                 info
    3.     storage.path              /var/log/flb-storage/
    4.     storage.sync              normal
    6.     storage.backlog.mem_limit 5M
    8.     name          cpu
    11. [INPUT]
    12.     name          mem
    13.     storage.type  memory