Log analytics

    The following image shows all of the components used for log analytics with Fluent Bit, Data Prepper, and OpenSearch.

    In the application environment, run Fluent Bit. Fluent Bit can be containerized through Kubernetes, Docker, or Amazon Elastic Container Service (Amazon ECS). You can also run Fluent Bit as an agent on Amazon Elastic Compute Cloud (Amazon EC2). Configure the Fluent Bit http output plugin to export log data to Data Prepper. Then deploy Data Prepper as an intermediate component and configure it to send the enriched log data to your OpenSearch cluster. From there, use OpenSearch Dashboards to perform more intensive visualization and analysis.

    Log analytics pipelines in Data Prepper are extremely customizable. The following image shows a simple pipeline.

    Log analytics component

    The accepts log data from Fluent Bit. This source accepts log data in a JSON array format and supports industry-standard encryption in the form of TLS/HTTPS and HTTP basic authentication.

    Processor

    Data Prepper 1.2 and above come with a . The Grok Processor is an invaluable tool for structuring and extracting important fields from your logs, making them more queryable.

    For more information about Grok features, see the documentation.

    There is a generic sink that writes data to OpenSearch as the destination. The has configuration options related to an OpenSearch cluster, like endpoint, SSL/username, index name, index template, and index state management.

    The following sections discuss pipeline configuration.

    Example pipeline with SSL and basic authentication enabled

    This example pipeline configuration comes with SSL and basic authentication enabled for the :

    This pipeline configuration is an example of Apache log ingestion. Don’t forget that you can easily configure the Grok Processor for your own custom logs. You will need to modify the configuration for your OpenSearch cluster.

    The following are the main changes you need to make:

    • hosts – Set to your hosts.
    • index – Change this to the OpenSearch index to which you want to send logs.
    • username – Provide your OpenSearch username.
    • password – Provide your OpenSearch password.
    • – If you use Amazon OpenSearch Service with AWS signing, set this value to the AWS Region in which your cluster is hosted.

    You will need to run Fluent Bit in your service environment. See for installation instructions. Ensure that you can configure the Fluent Bit http output plugin to your Data Prepper HTTP source. The following is an example fluent-bit.conf that tails a log file named test.log and forwards it to a locally running Data Prepper HTTP source, which runs by default on port 2021.

    The following is an example file without SSL and basic authentication enabled on the HTTP source:

    If your HTTP source has SSL and basic authentication enabled, you will need to add the details of http_User, http_Passwd, tls.crt_file, and tls.key_file to the fluent-bit.conf file, as shown in the following example.

    Example: Fluent Bit file with SSL and basic authentication enabled

    The following is an example file with SSL and basic authentication enabled on the HTTP source:

    Next steps

    See the Data Prepper Log Ingestion Demo Guide for a specific example of Apache log ingestion from FluentBit -> Data Prepper -> OpenSearch running through Docker.

    In the future, Data Prepper will offer additional sources and processors that will make more complex log analytics pipelines available. Check out the to see what is coming.

    If there is a specific source, processor, or sink that you would like to include in your log analytics workflow and is not currently on the roadmap, please bring it to our attention by creating a GitHub issue. Additionally, if you are interested in contributing to Data Prepper, see our Contributing Guidelines as well as our and plugin development guide.