Backpressure

In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restrict the amount of data than an input plugin can ingest, this is done through the configuration parameter Mem_Buf_Limit.

• Mem_Buf_Limit is set to 1MB (one megabyte)
• input plugin tries to append 700KB
• output plugin backend (HTTP Server) is down
• engine scheduler will retry the flush after 10 seconds
• input plugin tries to append 500KB

At this exact point, the engine will allow to append those 500KB of data into the engine: in total we have 1.2MB. The options works in a permissive mode before to reach the limit, but the limit is exceeded the following actions are taken:

• block local buffers for the input plugin (cannot append more data)

After some seconds if the scheduler was able to flush the initial 700KB of data or it gave up after retrying, that amount memory is released and internally the following actions happens:

• Upon data buffer release (700KB), the internal counters get updated
• Counters now are set at 500KB
• Since 500KB is < 1MB it checks the input plugin state
• If the plugin is paused, it invokes a resume callback

About pause and resume Callbacks

The plugin who implements and keep a good state is the plugin. When the pause callback is triggered, it stop their collectors and stop appending data. Upon resume, it re-enable the collectors.