Amazon S3

The Amazon S3 output plugin allows you to ingest your records into the cloud object store.

The plugin can upload data to S3 using the multipart upload API or using S3 . Multipart is the default and is recommended; Fluent Bit will stream data in a series of ‘parts’. This limits the amount of data it has to buffer on disk at any point in time. By default, every time 5 MiB of data have been received, a new ‘part’ will be uploaded. The plugin can create files up to gigabytes in size from many small chunks/parts using the multipart API. All aspects of the upload process are configurable using the configuration options.

The plugin allows you to specify a maximum file size, and a timeout for uploads. A file will be created in S3 when the max size is reached, or the timeout is reached- whichever comes first.

Records are stored in files in S3 as newline delimited JSON.

The plugin requires s3:PutObject permission.

In Fluent Bit, all logs have an associated tag. The s3_key_format option lets you inject the tag into the s3 key using the following syntax:

  • $TAG => the full tag
  • $TAG[n] => the nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with s3_key_format_tag_delimiters.

In the example below, assume the date is January 1st, 2020 00:00:00 and the tag associated with the logs in question is my_app_name-logs.prod.

With the delimiters as . and -, the tag will be split into parts as follows:

  • $TAG[0] = my_app_name
  • $TAG[2] = prod

So the key in S3 will be /prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz.

The store_dir is used to temporarily store data before it is uploaded. If Fluent Bit is stopped suddenly it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the store_dir for existing data and will try to send it.

There is one minor drawback to multipart uploads- the file and data will not be visible in S3 until the upload is completed with a CompleteMultipartUpload call. The plugin will attempt to make this call whenever Fluent Bit is shut down to ensure your data is available in s3. It will also store metadata about each upload in the store_dir, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the files will still be present on restart).

Using S3 without persisted disk

If you run Fluent Bit in an environment without persistent disk, or without the ability to restart Fluent Bit and give it access to the data stored in the store_dir from previous executions- some considerations apply. This might occur if you run Fluent Bit on AWS Fargate.

In these situations, we recommend using the PutObject API, and sending data frequently, to avoid local buffering as much as possible. This will limit data loss in the event Fluent Bit is killed unexpectedly.

The following settings are recommended for this use case:

  1. [OUTPUT]
  2.      Name s3
  3.      Match *
  4.      bucket your-bucket
  5.      region us-east-1
  6.      total_file_size 1M
  7.      upload_timeout 1m
  8.      use_put_object On

Fluent Bit 1.7 adds a new feature called workers which enables outputs to have dedicated threads. This s3 plugin has partial support for workers. The plugin can only support a single worker; enabling multiple workers will lead to errors/indeterminate behavior.

Example:

  1. [OUTPUT]
  2.      Name s3
  3.      Match *
  4.      bucket your-bucket
  5.      region us-east-1
  6.      total_file_size 1M
  7.      upload_timeout 1m
  8.      use_put_object On

If you enable a single worker, you are enabling a dedicated thread for your S3 output. We recommend starting without workers, evaluating the performance, and then enabling a worker if needed. For most users, the plugin can provide sufficient throughput without workers.

In order to send records into Amazon S3, you can run the plugin from the command line or through the configuration file.

The s3 plugin, can read the parameters from the command line through the -p argument (property), e.g:

In your main configuration file append the following Output section:

  1.      Name s3
  2.      Match *
  3.      bucket your-bucket
  4.      region us-east-1
  5.      store_dir /home/ec2-user/buffer
  6.      total_file_size 50M
  7.      upload_timeout 10m
  1. [OUTPUT]
  2.      Name s3
  3.      Match *
  4.      bucket your-bucket
  5.      region us-east-1
  6.      store_dir /home/ec2-user/buffer
  7.      use_put_object On
  8.      total_file_size 10M
  9.      upload_timeout 10m

Amazon distributes a container image with Fluent Bit and this plugins.

GitHub

github.com/aws/aws-for-fluent-bit

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

For example, you can pull the image with latest version by:

  1. docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

  1. aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the for more details.

Docker Hub

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

For more see .