Amazon CloudWatch

    The Amazon CloudWatch output plugin allows to ingest your records into the CloudWatch Logs service. Support for CloudWatch Metrics is also provided via .

    This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the aws/amazon-cloudwatch-logs-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named ; this new high performance CloudWatch plugin is called cloudwatch_logs to prevent conflicts/confusion. Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

    See for details on how AWS credentials are fetched.

    Getting Started

    In order to send records into Amazon Cloudwatch, you can run the plugin from the command line or through the configuration file:

    The cloudwatch plugin, can read the parameters from the command line through the -p argument (property), e.g:

    In your main configuration file append the following Output section:

    1. [OUTPUT]
    2.     Name cloudwatch_logs
    3.     region us-east-1
    4.     log_group_name fluent-bit-cloudwatch
    5.     log_stream_prefix from-fluent-bit-
    6.     auto_create_group On

    Example:

    1. [OUTPUT]
    2.     Name cloudwatch_logs
    3.     Match   *
    4.     region us-east-1
    6.     log_stream_prefix from-fluent-bit-
    7.     auto_create_group On
    8.     workers 1

    If you enable a single worker, you are enabling a dedicated thread for your CloudWatch output. We recommend starting without workers, evaluating the performance, and then enabling a worker if needed. For most users, the plugin can provide sufficient throughput without workers.

    Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. cloudwatch_logs output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above mentioned input plugins, cloudwatch_logs output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set json/emf as the value of log_format config option, CloudWatch will extract custom metrics from embedded JSON payload.

    Note: Right now, only cpu and mem metrics can be sent to CloudWatch.

    For using the mem input plugin and sending memory usage metrics to CloudWatch, we can consider the following example config file. Here, we use the aws filter which adds ec2_instance_id and az (availability zone) to the log records. Later, in the output config section, we set ec2_instance_id as our metric dimension.

    The following config will set two dimensions to all of our metrics- ec2_instance_id and az.

    2.     Match *
    4. [OUTPUT]
    5.     Name cloudwatch_logs
    6.     Match *
    7.     log_stream_name fluent-bit-cloudwatch
    8.     log_group_name fluent-bit-cloudwatch
    9.     region us-west-2
    10.     log_format json/emf
    11.     metric_namespace fluent-bit-metrics
    12.     metric_dimensions ec2_instance_id,az
    13.     auto_create_group true

    Amazon distributes a container image with Fluent Bit and these plugins.

    GitHub

    Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

    1. docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

    For example, you can pull the image with latest version by:

    If you see errors for image pull limits, try log into public ECR with your AWS credentials:

      You can check the Amazon ECR Public official doc for more details

      Docker Hub

      amazon/aws-for-fluent-bit

      Amazon ECR

      You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

      1. aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/